Rich Hickey: Deconstructing the Database - <AUTHOR> https://www.youtube.com/watch?v=Cym4TZwTCNU <comment><USER2> Watching this talk has so far (I'm halfway through, and now giving up) been very disappointing, primarily because many of the features and implementation details ascribed to "traditional databases" are not true of the common modern SQL databases, and almost none of them are true of PostgreSQL. As an initial trivial example, many database systems allow you to store arrays. In the case of PostgreSQL, you can have quite complex data types, from dictionaries and trees to JSON, or even whatever else you want to come up with, as it is a runtime extensible system.

However, it really gets much deeper than these kinds of surface details. As a much more bothersome example that is quite fundamental to the point he seems to be taking with this talk, at about 15:30 he seriously says "in general, that is an update-in-place model", and then has multiple slides about the problems of this data storage model. Yet, _modern databases don't do this._ Even _MySQL_ doesn't do this (anymore). Instead, modern databases use MVCC, which involves storing all historical versions of the data for at least some time; in PostgreSQL, this could be a very long time (when a manual VACUUM occurs; if you want to store things forever, this can be arranged ;P).

[http://en.wikipedia.org/wiki/Multiversion_concurrency_contro...](http://en.wikipedia.org/wiki/Multiversion_concurrency_control)

This MVCC model thereby directly solves one of the key problems he spends quite a bit of time at the beginning of his talk attempting to motivate: that multiple round-trips to the server are unable to get cohesive state; in actuality, you can easily get consistent state from these multiple queries, as within a single transaction (which, for the record, is very cheap under MVCC if you are just reading things) almost all modern databases (Oracle, PostgreSQL, MySQL...) will give you an immutable snapshot of what the database looked like when you started your transaction. The situation is actually only getting better and more efficient (I recommend looking at PostgreSQL 9.2's serializable snapshot isolation).

At ~20:00, he then describes the storage model he is proposing, and keys in on how important storing time is in a database; the point is also made that storing a timestamp isn't enough: that the goal should be to store a transaction identifier... but again, this is how PostgreSQL already stores its data: every version (as again: it doesn't delete data the way Rich believes it does) stores the transaction range that it is valid for. The only difference between existing SQL solutions and Rich's ideal is that it happens per row instead of per individual field (which could easily be modeled, and is simply less efficient).

Now, the point he makes at ~24:00 actually has some merit: that you can't easily look up this information using the presented interfaces of databases. However, if I wanted to hack that feature into PostgreSQL, it would be quite simple, as the fundamental data model is already what he wants: so much so that the indexes are still indexing the dead data, so I could not only provide a hacked up feature to query the past but I could actually do so efficiently. Talking about transactions is even already simple: you can get the identifier of a transaction using txid_current() (and look up other running transactions if you must using info tables; the aforementioned per-row transaction visibility range is even already accessible as magic xmin and xmax columns on every table).

<comment><USER3> I had no idea one could get a consistent read view across multiple queries within a transaction using most sql databases. That does poke a hole in a major benefit that I thought was unique to datomic, great to know!

However, I do think trying to setup a sql database to be able to query against any previous view of the world based on a transaction id as datomic allows wouldn't be as "simple" as you make it out to be.

<comment><USER2> The reason I claim this would be simple is that PostgreSQL is _almost_ already doing this. The way the data is stored on disk, every row has two transactions identifiers, xmin and xmax, which represent the transaction when that row was inserted the the transaction that row was deleted; rows, meanwhile, are never updated in place, so the old data stays around until it is deleted by a vacuum.

To demonstrate more tangibly how this works, I just connected to my database server (running PostgreSQL 9.1), created a table and added a row. I did so inside of a transaction, and printed the transaction identifier. I then queried the data in the table from a new transaction, showing that the xmin is set to the identifier of the transaction that added the row.

Connection 1:

        demo=> create table q (data int);
        CREATE TABLE
        demo=> begin; select txid_current();
        BEGIN
        189028
        demo=> insert into q (data) values (0); commit;
        INSERT 0 1
        COMMIT
        demo=> begin; select xmin, xmax, data from q;
        BEGIN
        189028|0|0

Now, while this new transaction is still open, from a second connection, I'm going to create a new transaction in which I am going to update this row to set the value it is storing to 1 from 0, and then commit. In the first connection, as we are still in a "snapshot" (I put this term in quotes, as MVCC is obviously not copying the entire database when a transaction begins) from a transaction started before that update, we will not see the update happen, but the hidden xmax column (which stores the transaction in which the row is deleted) will be updated.

Connection 2:

        demo=> begin; select txid_current();
        BEGIN
        189029
        demo=> update q set data = 1; commit;
        UPDATE 1
        COMMIT
        demo=> select xmin, xmax, data from q;
        189029|0|1

Connection 1:

        demo=> select xmin, xmax, data from q;
        189028|189029|0

As you can see, the data that the other transaction was referencing has not been destroyed: the old row (the one with the value 0) is still there, but the xmax column has been updated to indicate that this column no longer exists for transactions that began after 189029 committed. However, at the same time, the new row (with the value 1) also exists, with an xmin of 189029: transactions that begin after 189029 committed will see that row instead. No data was destroyed: and this data is persisted this way to disk (it isn't just stored in memory).

My contention then is that it should be a fairly simple matter to take a transaction and backdate when it began. As far as I know, there is no reason that this would cause any serious problems as long as a) it was done before the transaction updated or inserted any data, b) there have been no vacuums during the backdated period, c) HOT (heap-only tuple) updates are disabled (in essence, this is an optimization designed to do online vacuuming), and _maybe_ d) the new transaction is read only (although I am fairly confident this would not be a requirement).

For a more complete implementation, one would then want to be able to build transactions (probably read-only ones; I imagine this would cause serious problems if used from a writable transaction, and that really isn't required) that "saw all data as if all data in the database was alive", which I also believe would be a pretty simple hack: you just take the code that filters dead rows from being visible based on these comparisons and add a transaction feature that lets you turn them off. You could then use the already- implemented xmin and xmax columns to do your historical lookups.

P.S. BTW, if you want to try that demo at home, to get that behavior you need to use the "repeatable read" isolation level, which uses the start of the transaction as the boundary as opposed to the start of the query. This is not the default; you might then wonder if it is because it is expensive and requires a lot more coordination, and as far as I know the answer is "no". In both cases, all of the data is stored and is tagged with the transaction identifiers: the difference is only in what is considered the reference time to use for "which of the rows is alive".

However, it does mean that a transaction that attempts to update a value that has been changed from another transaction will fail, even if the updating transaction had not previously read the state of the value; as most reasonable usages of a database actually work fine with the relaxed semantics that "data truly committed before the query executes" provides (as that still wouldn't allow data you update to be concurrently and conflictingly updated by someone else: their update would block) and those semantics are not subject to "this transaction is impossible" errors.

Both Connections (setup):

        demo=> set session characteristics as transaction isolation level repeatable read;
        SET

<comment><USER3> Thanks for taking the time to elaborate, very interesting. I wonder if the sql db vendors or open source projects will take the next step to make querying against a transaction ID possible given the underlying implementation details bring it pretty close.

I also see Rich has made some interesting points elsewhere in this thread about consistent views being available outside of transactions and without need for coordination (within datomic) - seems more appropriate to comment directly there though.

Overall I think it's important to understand these nuances, and not view datomic as some revolutionary leap, even if I am excited about the project. I appreciate your insight into the power already within sql db engines.

<comment><USER2> I am not certain whether your response comes from a reading of my comment before or after the paragraph I added that started with "for a more complete implementation", but if it was from before I encourage you to read that section: the ability to do the query is pretty much already there due to the xmin/xmax fields that PostgreSQL is already reifying.

</comment></comment></comment></comment></comment><comment><USER4> There are two people that I will stop what I'm doing and watch every new lecture they make: Rich Hickey and Bret Victor. Both are visionaries.

<comment><USER5> And both will be at Strange Loop this year! <http://thestrangeloop.com/sessions>

<comment><USER6> Strange Loop is such a great conference. I wish there was something like that here in Europe.

<comment><USER5> What about Devs Love Bacon? <http://devslovebacon.com/>

</comment></comment></comment></comment><comment><USER7> Fascinating stuff. Some things that came up for me while watching this and the other videos on their site[1]:

It's not Open Source, for anyone who cares about that. It's interesting how strange it feels to me for infrastructure code to be anything other then Open Source.

I'm sort of shocked that the query language is still passing strings, when Hickey made a big deal of how the old database do it that way. I guess for me a query is a data structure that we build programmatically, so why force the developer to collapse it into a string? Maybe because they want to support languages that aren't expressive enough to do that concisely?

[1] <http://www.datomic.com/videos.html>

<comment><USER8> You are not forced to pass strings. That's merely a convenience (for Java). You can construct and pass data structures instead.

<comment><USER9> Yes, and in clojure you acctually use data literals not just strings.

</comment></comment></comment><comment><USER10> I'm always puzzled when the Datomic folks speak of reads not being covered under a transaction. This is dangerous.

Here's the scenario, that in a conventional update-oriented store, is termed as a "lost update". "A" reads object.v1, "B" reads the same version, "B" adds a fact to the object making it v2, then "A" comes along and writes obj.v3 based on its own _stale_ knowledge of the object. In effect, it has clobbered what "B" wrote, because A's write came later and has become the latest version of the object. The fact that DAtomic's transactor serialized writes is meaningless because it doesn't take into account read dependency.

In other words, DAtomic gives you an equivalent of Read-committed or snapshot isolation, but not true serializability. I wouldn't use it for a banking transaction for sure. To fix it, DAtomic would need to add a test-and-set primitive to implement optimistic concurrency, so that a client can say, "process this write only if this condition is still true". Otherwise, two clients are only going to be talking past each other.

<comment><USER11> Addressed here: <http://news.ycombinator.com/item?id=4448351>

<comment><USER10> Lovely. That addresses my concern.

</comment></comment></comment><comment><USER12> Anyone understands how this system would deal with CAP theorem, in the case of a regular "add 100$ then remove 50$ to the bank account, in that order and in one go" type of transaction ? The transactor is supposed to "send" novelty to peers, so that they update their live index. That's one point where i would see trouble (suppose it lags, one "add" request goes to one peer, the "read" goes to the second, you don't find what you just add...) Another place i see where it could mess things up is the "Data store" tier, which uses the same traditional technics as of today to replicate data between different servers (one peer requests facts from a "part" of the data store that's not yet synchronized with the one a second peer requests). It seems like all those issues are addressed on his "a fact can also be a function" slide, but he skips it very quickly, so if anyone here could tell me more...

<comment><USER13> Well there is _one_ transactor, the transactor handles each transaction sequential, so the transactor can abort a transaction when the world has changed since it got queued to the transactor.

There is (virtually) infinite read scalability. Each datum has a time associated with it, so you might not see the latest information (yet), you know the state of the world at a certain point in time.

I think it 's a really well designed system.

</comment></comment><comment><USER14> I recall Datomic making a bit of a splash on HN when it was announced 6+ months ago, but basically crickets since then. Anybody build something cool that took advantage of Datomic's unique design?

<comment> Tuna-Fish People tend to be more conservative about data than they are about the other parts of their stack -- probably for a good reason.

I don't think datomic (or it's kin) will have that huge of an influence until years from now.

<comment><USER15> I think Datomic is potentially disruptive and represents some great thinking on the part of an individual. Whether it will be disruptive will hinge on how well that thinking has subsumed the years of industry experience and practicalities, not to forget the conservative approach to data. I'd be interested to see how it pans out.

<comment><USER9> I think if will also be imortend what other databases come out of this. There is space for other prioritys in terms of CAP in a world were perseption and process are seperated. Or just other implmentations of the same ideas, opensource maybe.

</comment></comment></comment></comment><comment><USER16> Anyone have a summary for those of us who don't want to watch an hour-long video?

<comment><USER17> I'm only a few minutes into it, but so far the tl;dr is:

"There are lots of things we do with respect to databases simply because of the limitations of our databases. Here are some things that could be made far simpler if you just had a better database."

Yes it's partly a sales presentation for Datomic, but you could do worse than be sold by Rich Hickey.

<comment><USER17> More notes on the video:

\- Rich's whole view on the world is pretty consistent with respect to this talk. If you know his view on immutability, values vs identity, transactions, and so forth, then you already have a pretty good idea about what kind of database Rich Hickey would build if Rich Hickey built a database (which, of course, he did!)

\- The talk extends his "The Value of Values" keynote [1] with specific applicability to databases

\- Further, there is an over-arching theme of "decomplecting" a database so that problems are simpler. This follows from his famous "Simple made easy" talk [2]

\- His data product, Datomic, is what you get when you apply the philosophies of Clojure to a database

I've talked about this before, but I still think Datomic has a marketing problem. Whenever I think of it, I think "cool shit, big iron". Why don't I think about Datomic the same way I think about, say, "Mongodb". As in, "Hey, let me just download this real quick and play around with it!" I really think the folks at Datomic need to steal some marketing tricks from the NoSQL guys so we get more people writing hipster blog posts about it ;-)

[1] <http://www.infoq.com/presentations/Value-Values>

[2] <http://www.infoq.com/presentations/Simple-Made-Easy>

</comment></comment></comment><comment><USER18> this reminds me a lot of "How to beat the CAP theorem" <http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html>

</comment><comment><USER19> How would you idiomatically fix invalid data in Datomic? for example, if you needed to update a badly entered value in a record, but keep the record's timestamp the same so as not to screw up historical queries?

<comment><USER20> You wouldn't. You can't change history either. You would insert a compensating transaction and the data would be fixed from that time forward when you inserted the compensating transaction.

<comment><USER19> Figures. So I guess that, in the case where you both need to worry about fixing invalid data, and also need to do historical queries, you would have to add your own timestamp to the data to represent the actual time of the event, because the built-in timestamp is just giving you the time of the state of the database. Hopefully that wouldn't get too hairy.

</comment></comment></comment><comment><USER21> I often wonder, is Phd in computer science really required to do awesome work?

<comment><USER22> Rich Hickey never went to school for Computer Science as far as I know. He studied music composition.

<comment><USER23> Rich has a CS master's degree.

<comment><USER22> Did not know that!

</comment></comment></comment></comment><comment><USER24> I'm getting "This video is currently unavailable"?

<comment><USER25> I don't know - are you?</comment></comment>


Prof. Matthew Green asked to remove NSA post by JHU dean - <AUTHOR> https://twitter.com/matthew_d_green/status/377120854678585344 Here's the post while it's still up: http://blog.cryptographyengineering.com/2013/09/on-nsa.html <comment><USER2> The post:

[http://blog.cryptographyengineering.com/2013/09/on- nsa.html](http://blog.cryptographyengineering.com/2013/09/on-nsa.html)

</comment><comment><USER3> Ordering a professor to remove a post about his area of research from a university server is an excellent way to get that post much more widely covered in the press. Great move, NSA/JHU.</comment>


Developer Survey Results - <AUTHOR> https://insights.stackoverflow.com/survey/2018 <comment><USER3> Cool to see Redis to be in top position. For what is worth, Redis confessed to love developers, so it is definitely a reciprocal feeling.

Btw surprised to see "contributing to OSS" at ~50%. Looks like an incredibly high number (cit. Trump), so would be awesome to see this question more fine grained next time. This may be an indirect measure of all the work that goes into OSS to make it sustainable.

I would like to see: kind of contribution (Opening issues, PR, managing a project, ...) and average hours spent per week.

<comment><USER4> Definitely deserved for Redis, so thanks to you!

And I completely agree about the OSS - this makes me _really_ question the sampling bias of this whole survey, because there's no way so many developers contribute to OSS. Unless you really do mean something like "reporting issues", etc, in which case i'd _still_ guess the real number was below 10%.

<comment><USER5> Maybe "have a GitHub repo gathering dust" already counts?

</comment></comment></comment><comment><USER6> I love the part about SO developer story: 84% either do not use it or have never even heard about it (which basically shows its failure) yet insights team tries to interpret it as 40% actually using it. Based only on this point, I wonder how extremely biased their conclusions are overall.

<comment><USER7> Quite a few of those stats seem suspect. For example, I personally find it hard to believe that "80% of professional developers" prize Javascript above all other languages. Seems unlikely based on my network (which of course is a representative sample of all developers).

<comment><USER8> Where did you get the prized by 80% from?

I only see the "Most Popular Technologies" and it has JavaScript at 71% and it's not "prized":

[https://insights.stackoverflow.com/survey/2018#most- popular-...](https://insights.stackoverflow.com/survey/2018#most-popular- technologies)

As far as it's popularity is concerned. It's not surprising the in least. If you have anything to do with web development you're going to have to use JavaScript. Making it "popular" out of necessity but not "prized".

<comment><USER7> I'm using "prized" here as a synonym of "favored"; perhaps it's a bit archaic. I don't intend to alter or misconstrue the meaning of the survey result.

Certainly, a front-end developer has a choice of compile-to-JS languages from which they can choose.

</comment></comment></comment></comment><comment><USER9> With all the talk of Vim vs. Emacs, it surprises me that Emacs is in the low single digits, while Vim is 25% of all respondents.

It does not surprise me that it is 40% (and #1) amongst sysadmin/devops practitioners.

<comment> cup-of-tea That's because vim and emacs have different purposes and the choices were not exclusive. I use emacs and even I know enough vim to get around and use it regularly. 25% of people claim to use vim, but for what? Occasional editing of individual files? In contrast, those emacs users almost certainly use emacs for _everything_.

<comment> purple-again This exactly, the guy next to me said yes to Vim (and makes sure to talk shit to me literally every single time I open nano) but he literally used Atom all day long and only uses Vim when editing a file stored on a remote server.

I don't care how you feel about nano, I use it less than .0000000000001% of my development time and it serves its purpose just fine.

<comment><USER10> Assuming you work 80,000 hours in your career:

80000 _(.0000000000001 /100)_60*60 = 2.88e-7 seconds spent in nano in your lifetime

[ I know it's an exaggeration :^) ]

</comment></comment></comment></comment><comment><USER11> Note the salary graph that shows if you've got over 10 years experience you can either be an engineering manager (small cog in a big machine) and make a ton of money, or a CTO (big cog in probably a much smaller machine) and make 20% less.

<comment><USER12> the CTO/CEO point probably reflects that people who start their own company AND take the SO developer survey always call themselves CTO or CEO. People don't start their own company and call themselves a developer or an engineering manager. CTO ... of what? Most startups will die before they ever have an engineering manager.

<comment><USER13> C-suite is by definition close to the top but I wish more people would understand it also needs to be _far from the bottom_. You're not working on the types of things the C-suite works on unless you have a lot of people below you taking care of everything else.

</comment></comment></comment><comment><USER14> > Most developers feel that management is ultimately most accountable for > unethical results of code. Just under 20% of respondents said that a > developer who writes code used for unethical purposes is most responsible.

This kind of surprised me. On the one hand, we complain about being considered code monkeys and mere "implementors," but on the other hand, if we're asked to code something unethical, well "we're just following orders." I personally think we, as a profession, need to kind of grow a backbone and step up our responsibility to no do harm. Ask doctors this survey question and see the wildly different results.

<comment><USER15> I think that one result is a bit misleading.

79.6% said they consider the ethical implications of what they are asked to do.

82.3% said they would report ethical concerns in some form.

58.5% said they would refuse to write code they thought was not ethical (36.6% said they are unsure).

<comment><USER14> Good point. I was also puzzled by that result, given the responses to the question I mentioned. The picture it paints is that software professionals feel their duty is to consider, complain, and (less so) refuse, but ultimately responsibility lies elsewhere.

</comment></comment></comment><comment><USER16> As a queer female go programmer with over 15 years of programming experience, I did not realize how much of a statistical outlier I am. Neat. (I have trouble adopting the mantle of minority, as a well-paid middle-aged white person).

<comment><USER17> As a straight male Kotlin wannabe with over 35 years of programming experience (with the first 20+ years paired with hardware development), I'm also going to refuse to be categorized (nothing wrong with Cobol but I like the new shiny things the youngin's are playing with). I think the most interesting facet of software (the cerebral creation of something out of nothing) is that there's a way for everyone to contribute in a way they find enjoyable.

</comment></comment><comment><USER18> I was seriously surprised how popular VSCode has become. I know it's good, but for MS to win IT crowd that fast... kudos.

<comment><USER19> Visual Studio Code + the cquery extension is heaven for me coding in C/C++.

<comment><USER20> Also checkout Clang-Format. It's so awesome to mark some unintended code and have it formatted instantly in your editor with a key press. All while editing the file via the Git diff view :)

</comment></comment></comment><comment><USER21> Remember that this data is representative of Stack Overflow's audience, not of the entire industry.

<comment><USER22> I'd say that a pretty representative swath of the industry visits StackOverflow. What segment of the industry do you think is missing in this representation?

<comment><USER21> Different sectors of the industry use StackOverflow in varying amounts. Systems programmers on the whole probably use SO a lot less than JavaScript programmers, for example. More senior engineers use it less than more junior engineers. And so on.

</comment></comment></comment><comment><USER23> I'm actually surprised that around 64% of the developers are having a "Computer science, computer engineering, or software engineering" degree.

I do my personal statistics for colleagues I've met and the percentage having no degree or degree in another field is around 7 out of 10.

Maybe that's because I primarily work for startups and do my survey in the startup culture, but still it's fascinating how much of those developers took the decision to become one at around high school age.

<comment><USER24> This probably depends largely on the area. Since you mention high school, I guess you're mostly referring to the US situation. From what I've seen in Europe, most (I'd say more than half) of the devs actually do have some degree in CS, or at did a corresponding apprenticeship.

But still, I'd say the percentage of career changers is much higher than in most other fields.

</comment></comment><comment><USER25> Two unrelated remarks:

I'm surprised that Elixir was the 6th most loved language last year and disappeared this time, while Erlang gained some points. Was it even in the survey?

Also it's interesting that women placed compensation at 4th in "Differences in Assessing Jobs by Gender", but if you point that fact as a contributing factor for the pay gap you can be labeled as sexist.

<comment><USER26> > Also it's interesting that women placed compensation at 4th in "Differences > in Assessing Jobs by Gender", but if you point that fact as a contributing > factor for the pay gap you can be labeled as sexist.

How can you be labeled sexist if you point out a simple fact?

<comment> atom-morgan Ask Damore.

<comment><USER27> Yes, he clearly only pointed out facts and nothing else. This is a fair and honest assessment of what happened.

</comment></comment></comment></comment><comment><USER28> What's up with Rails not being included in the Frameworks section when Django and Node.js are? According to Google Trends, Django and Rails have similar interest:

[https://trends.google.com/trends/explore?q=%2Fm%2F0505cl,%2F...](https://trends.google.com/trends/explore?q=%2Fm%2F0505cl,%2Fm%2F06y_qx,%2Fm%2F0bbxf89)

<comment><USER29> That's weird. Also, can Node.js even be considered a framework? Isn't it just a run-time system?

<comment><USER30> This illustrates the confusion about JS in general.

</comment></comment></comment><comment><USER31> Rust wins "Most loved" for the third year in row!

<comment><USER32> And yet corporates still don't use it and schools don't teach it.

This shows how much power opinions have.

<comment><USER33> There are over 100 companies who are willing to put their logo on our website, and others we know about but can't say, including Fortune 500 companies. Dropbox just went public and their project in Rust (and Go) was cited as a reason for their increased profitability. There have also been multiple university classes in Rust.

It's a start.

(Also, this survey never claimed to be measuring either of those metrics...)

<comment> purple-again We have been using Rust in production for around 2 years now (yes its been a bumpy ride...) at a big stuffy old school finance company that I'm sorry doesn't want their logo on your website.

The Rust code is not only mission critical, it is literally THE defining factor that has won us several very large contracts now.

I make no attempt to compare Rust to other alternatives in its class, we chose it because we were developing primarily with Ruby/Ember at the time and it felt like a natural fit.

<comment><USER33> Glad to hear it!

</comment></comment></comment></comment></comment><comment><USER34> I quite enjoy the irony that Coffeescript—a language apparently invented to make Javascript more palatable—is the third most dreaded language in the survey.

<comment> purple-again Atom is written entirely in Javascript and I can edit it however I want??? Thats AWESOME!

Wait its written in coffescript...fuck you.

That's a replay of a real life experience that went down when I first heard of Atom.

<comment><USER35> The more layers you add, the less inclusive it becomes. Coffeescript is decidedly a divisive technology.

</comment></comment></comment><comment><USER36> I think it's interesting how VS Code jumped from 5th place among web developers in 2017 [1] to first place [2]. This made Sublime Text lose a place, putting it in third. If I were ST's Jon Skinner, I would not be happy about having M$ as a (free!) competitor.

[1]: [https://insights.stackoverflow.com/survey/2017#technology- mo...](https://insights.stackoverflow.com/survey/2017#technology-most-popular- developer-environments-by-occupation)

[2]: [https://insights.stackoverflow.com/survey/2018/#technology-m...](https://insights.stackoverflow.com/survey/2018/#technology- most-popular-development-environments)

</comment><comment><USER37> There seem to be a lot more young developers than old developers. But, why? I can only assume that either, the number of developers is increasing (more and more young people entering the work force), or a lot of people stop being software engineers after 8 years. People usually work between the ages 22 and 65, so given a even distribution, if the industry isn't growing by leaps and bounds, the average work experience should eventually be about 20 years with an average age of 42.

<comment><USER38> Growth of technology and an increase in job openings. Plus, a bit of cultural shift too. I think a lot of older developers grew up during a time when spending a lot of time using technology was considered weird. While today, not using technology all the time is what is weird.

Also, software developers are much more highly regarded by young people now than ever before. There's this image of the young, rich entrepreneur creating products that everyone loves, and people want to be this. So a lot more smart people get into the industry because it's the cool place to be, hence the rise of "brogrammers."

</comment></comment><comment><USER39> I am not sure about you, but as my career as a developer progressed I rely less on Stack Overflow today as I did in the past. To me it seems that this survey may have a strong bias.

</comment><comment><USER40> One nitpick here - they refer to Angular as AngularJS - these are actually two different things. I don't quite remember but did the survey itself distinguish between the two? If not then this will skew the numbers upwards.

<comment><USER41> i rather they be combined. I know they are different but one is the successor of the other.

</comment></comment><comment><USER42> I don't believe that "Almost half of professional developers contribute to open source projects.", even if you count every single half baked unfinished project on Github as contributing to open source.

<comment><USER43> Contributions include bug submissions and (were possible) comments in RFCs, right?

<comment><USER42> Yes, but even there the 50% seems too much. I can see people submit that much bugs in cutting edge communities (where there are still a lot of new bugs to submit), but when you work with something stable you rarely ever need to do so. There is huge amount of positions where people successfully use what exists.

Plus, calling bug submissions as open source contribution is a stretch. You would not call "contributing to close source" when someone reports a bug to a company.

</comment></comment></comment><comment><USER44> The "Platforms" answers are hard to believe, or let's say it's hard for me to believe that they are remotely representative of the overall developer population.

18% claim to have done development work for macOS. That's more than for iOS, which is neck and neck with Raspberry Pi.

What's going on here? Have web devs simply checked all platform boxes?

<comment><USER25> It's also really strange (to me) how no PHP frameworks or CMS made the cut.

<comment><USER44> WordPress is up there at 16%

<comment><USER25> Ah yes, I was more thinking about the "Frameworks, Libraries, and Tools" section, my bad.

</comment></comment></comment></comment><comment><USER45> I'm surprised Vue doesn't show up on the list anywhere. From everything I've seen Vue should probably show up in either the loved or wanted categories.

<comment><USER46> I'm not sure anybody outside of HN readership knows about Vue at all. It's either Angular, old Angular, or React everywhere.

<comment><USER47> Don't you think that's a bit naïve? Take this anecdotal counterevidence for instance: none of the four people I work with visit HN but all know of Vue.

</comment></comment></comment><comment><USER48> Who Is Ultimately Most Responsible for Code That Accomplishes Something Unethical? 19.7% voted "the developer who wrote it"

Do Developers Have an Obligation to Consider the Ethical Implications of Their Code? 79.6% voted "Yes"

<comment><USER49> I initially bumped on this one too, but "obligation to consider" doesn't mean "obligation to quit their jobs over". You consider the ethical implications, you push back against management, management tells you sit down. Are you now obligated to quit? If you decide to keep your job, knowing that the code will be written either way, are you now ultimately responsible for it?

( _Completely_ unrelated but the first thing that came to mind: [https://www.youtube.com/watch?v=GKdgDX2dGh0](https://www.youtube.com/watch?v=GKdgDX2dGh0))

</comment></comment><comment><USER24> I wonder why there's so much dread for Cordova. It's a framework you don't interact with often while using it, so I would assume devs would have neutral, unemotional stance towards it, but apparently not.

<comment><USER50> Because a lot of us web deva used it once for a contract before swearing it and all of mobile development off forever. I'm sure it's come a long way since, but it was pretty brutal to understand and debug years ago.

<comment><USER5> Because you're actually interacting with two completely different stacks at least that have a pretty leaky abstraction. So you only get the painful parts of iOS development, certificates and build chains not the fun parts. Repeat that for Android. _points double barreled shotgun at chin_

As a native iOS dev I write my own wrappers for web applications, I think Cordova is too just too much hassle for what it really does and only limits my possibilities as a native developer hosting a web application.

<comment><USER51> Off-topic: Any good point in how make the android webview to perform well?

</comment></comment></comment></comment><comment><USER52> Wow people actually use Ocaml at work?

<comment><USER53> Yep! Facebook and Docker among them.

<comment><USER54> Also Jane Street etc.

</comment></comment></comment><comment><USER55> Another #FunFact: Perl ranked number 5 in "What Languages Are Associated with the Highest Salaries Worldwide?"

[https://insights.stackoverflow.com/survey/2018/#most- loved-d...](https://insights.stackoverflow.com/survey/2018/#most-loved- dreaded-and-wanted)

<comment><USER56> #1 in Perl programmer happiness also. (Not part of survey.)

:)

</comment></comment><comment><USER57> Just out of curiosity, why do so many people want to learn Go despite it not being used that much, 7.1% from the survey?

<comment><USER58> If you come from a Python/Ruby/JavaScript background (which is a lot of people), Go is a language that is not much more complex but will run faster and be easier to distribute. I can see that being a big appeal, especially compared to something like Rust or C++ which is going to require a lot more learning to become productive.

</comment></comment><comment><USER59> Whhy is the salary for F# developers high?

<comment><USER60> On that note: Anyone know of any good F# learning resources (preferably books/text) for an experienced C# developer who has only done a little bit of functional programming (Haskell, Clojure) here and there?

<comment><USER46> Get Programming with F# is finally done, and looks pretty good

[https://www.manning.com/books/get-programming-with-f- sharp](https://www.manning.com/books/get-programming-with-f-sharp)

Real-World Functional Programming With examples in F# and C# is another good one

[https://www.manning.com/books/real-world-functional- programm...](https://www.manning.com/books/real-world-functional-programming)

The NoStarch Book of F# is a pretty decent beginner's book

[https://nostarch.com/fsharp](https://nostarch.com/fsharp)

Would second the vote for F# Deep Dives as well.

</comment></comment></comment><comment><USER61> Coming from the web world I find the term "back end developer" a heavily abused term. Most developers of web products would consider themselves back- end developers and typically program in either Java (or one of its variants) or C#. This isn't back-end merely because it isn't in a browser.

Your average web product Java developer is all middleware. They write code that gels between internal services and output to the user environment. At most large web companies real back-end teams are heavily isolated and don't let the middleware people come anywhere close to them even if they are all writing in the same language. If your web-product Java developer needs data they call to a service managed by an unrelated team.

</comment><comment><USER62> Any idea why salaries for C/C++ developers seem to be way below average in these stats?

<comment><USER63> International responses skew the salary data and make it useless.

<comment><USER64> This is true. It always feels weird answering these kinds of questions living outside of the US. You can easily have a salary that is several times the average of your country (I'm talking like 20x) and still make about half of what you'd make in the US, when shown in USD.

</comment></comment></comment><comment><USER65> The "Why Do Developers Participate in Hackathons?" question is missing an answer option that I suspect is distressingly common: employer-mandated hackathon participation.

Multiple places I've worked, and many places I've heard about from my peers, require employees to participate (either as a "this is just mandatory" requirement or as a "this influences a milestone on your performance review" requirement) in company-sponsored or external hackathons.

This trend is somewhat distressing and deserves to be highlighted, I think.

</comment><comment><USER66> I like how Scala is connected to Spark and Hadoop but nothing else...

</comment><comment><USER67> >.NET Core

Thanks they hadn't put a Silverlight instead of .NET. StackOverlow makes this mistake third year in a row. It is not .NET Core, it is .NET.

.NET Core is just yet another implementation of .NET runtime.

<comment> james-skemp As a .NET developer, there's a difference between saying .NET and .NET Core, however.

You can be interested in .NET Core but not in .NET in general if you're looking for cross-platform applications, for example.

Having used .NET for almost a decade, I think I selected .NET Core for a couple options since I hadn't been able to learn about the differences. And there's pretty significant differences, especially tooling support (like the inability to really do database-first as easily as you can with .NET).

So, I think it's fine.

</comment></comment><comment><USER68> Nice to see C++ on the loved languages section, while C only deserved an entry on the dreaded ones above C++ and has half of the wanted percentage.

<comment><USER69> Other than the verbosity the new versions of C++ are so much nicer to work with than C++03. C++17 is even better, constexpr if simplifying templates, structured bindings (though I wish there was an ignore option), optionals, if the C++ standards committee isn't careful they might end up with an enjoyable language in a decade or so.

</comment></comment><comment> cup-of-tea I wonder how many are "web developers". They seem to have broken down that category into three or four possibly overlapping subsets. "Front-end developer" and "back-end developer" should be mutually exclusive because people in the intersection would be "full-stack developers". But if that were the case then more than 95% are web developers.

<comment><USER70> Backend does not necessarily mean web technology. I'm reminded of an oft-lost distinction between "The (Inter)net" and "The Web".

<comment> cup-of-tea I've not heard of other network services using the term "backend". Is writing a mail server "back end" development?

<comment><USER70> Yes. A rather uninspiring appraisal of cyberspace — web browsing & emailing — though.

</comment></comment></comment></comment><comment><USER37> It looks like most people stay at a job for just a year or 2. But, it would have been interesting to see a question as to why they left: were they forced out due to something the company did, or did they leave of their own accord for a higher salary (that was top ranked for how they assess a job)?

</comment><comment><USER58> Results aside, I really dislike the presentation of different data sets in separate tabs. If I want to see the difference between all developers and professionals, or between genders, I have to click back and forth when it could have all been presented in one chart to begin with.

</comment><comment><USER71> In the section "Valuing Compensation and Benefits by Geography", I was surprised that Germany (13.7%) and Canada (9.7%) showed similar results to the USA (11.5%) on the relative value of health insurance as part of compensation. I thought they had that covered.

</comment><comment><USER72> Do people not know that there is a database called Hana? Honestly before my time at SAP I also didn't know but I thought it's my personal failing.

</comment><comment><USER73> Was very surprised about the gender ratio: 92.7% Male, 6.8% Female, 0.9% Non- binary, 0.7% Transgender

<comment><USER74> What were you expecting?

<comment><USER73> A lot more females?

<comment><USER74> Like 10% more or like 40% more females?

I get that a lot of people want more diversity in Tech, but honestly it's just not something most girls are interested in.

If you saw the following gender ratios for pre-shcool teachers: 92.7% Female, 6.8% Male, 0.9% Non-binary, 0.7% Transgender

Would you be surprised? Would you think more men need help "getting into the industry" since they are underrepresented? Or would you think that most guys just don't want to do that for a job/career?

</comment></comment></comment></comment><comment><USER75> FTA :

> Race and Ethnicity > Sexual Orientation

Could we agree that : 1/ a coder is a coder, no matter his color, his sexual orientation, his whatever 2/ the notion of "race" is problematic at best

And before I get down voted : who cares about a statistic about sexual orientation or ethnicity in a survey about IT ? I mean honestly.

<comment><USER11> > who cares about a statistic about sexual orientation or ethnicity in a > survey about IT

People trying to make the IT industry accessible to all regardless of sexual orientation or ethnicity, who want to check in on how we're doing at that?

<comment><USER76> If you hire 50 random people (uniform distribution) the probability to get twice (or more than twice) as many employees of one sex as the other sex is about 3.5%. If you add ethnicity, age, sexual orientation etc. to the mix you will rarely get a random sample with all attributes proportional to their occurrence in population. Without quotas it will be really hard to be accessible to all regardless of [a long and ever expanding list of attributes].

<comment><USER75> Thanks for that answer !

</comment></comment></comment></comment><comment><USER77> What are people finding so dreadful about Redshift?

<comment><USER78> I think I can answer that, given we're in the business of Redshift performance.

Redshift is incredibly powerful, but can be a pain to work with if you're a data engineer. To get the most out of it, you need to turn a lot of knobs - and knowing how to tune it is really complex. But if you get it right - <USER2> blows everything else out of the water. Including Snowflake and BigQuery.

The key to success is using Redshift's Workload Management, and using best practices for your schema, table and query design. Many people who start using Redshift come from a mySQL world and are not familiar with a columnar storage db. Sort / dist keys are not concepts that people are necessarily familiar with.

</comment></comment><comment><USER79> Good!

</comment><comment><USER80> Quick question: How are ~14% of all respondents from India, but 4.1% categorized as Middle Eastern in the Race and Ethnicity breakdown?

This is just one example of a strange anomaly in the data.

<comment><USER81> I guess most will identify as 'South Asian'.

Also you can see that the race/ethnicity question only has ~55k responses out of 100k respondents in general.

To many of us it feels like a very US-specific question that we are glad to skip. Racial identity is just not part of our lives. I might identify as as European, but certainly not as 'White or of European descent'

<comment><USER82> Yeah, it is a bit weird to use US specific notions of race outside of the US itself. How many Europeans in Europe feel that of "European descent" is an accurate description?

</comment></comment></comment><comment><USER83> It's a little disappointing to see so many developers--maybe on average the most comfortable job in the world right now--prioritizing comforts like compensation, using their favorite languages, or remote work, over helping to improve the world.

Over 45% of men prioritized one of the above three, whereas a mere 6.5% prioritized "How widely used or impactful the product or service I'd be working on is." Revealingly, "How much impact _I_ could have by working there" isn't even an option. Sounds like the survey authors didn't even consider that people might actively care about improving the world.

So many developers have comfort to spare! Maybe spend a little bit more optimization power on trying to accomplish something important?

<comment><USER84> >prioritizing comforts like compensation, using their favorite languages, or remote work, over helping to improve the world.

Compensation isn't a comfort -- it's more like payment which is necessary. Not really something a developer could forgo.

Regarding languages -- why would they not want to leverage their knowledge to get more compensation?

<comment><USER85> > Compensation isn't a comfort

Sure it is. If you can "be fine" with $X, but are looking for a job that pays $Y (where $Y>$X), then that's seeking comfort.

<comment><USER86> $X directly relates to when I retire, so I see a high salary as buying time and freedom</comment></comment></comment></comment>


Opera 10.50 out for Mac, impressive performance and more - <AUTHOR> http://ajaxian.com/archives/opera-10-50 <comment><USER2> I've been using it since it came out for Windows a week or two ago, and it's wonderfully fast (as well as beautiful). Using Opera used to feel like sacrificing performance for functionality, but now I feel like I have the best of both worlds. It is still beta, and there are a couple bugs, but for the most part it's a great experience.

<comment><USER3> > sacrificing performance for functionality

That's what I feel about Firefox compared to Chrome or Opera.

<comment><USER2> Which Firefox extensions would you miss if you used Opera? Opera has built in support for tab thumbnails, mouse gestures, Greasemonkey scripts, widgets, etc.

<comment><USER4> Tree-style tabs:

<https://addons.mozilla.org/en-US/firefox/addon/5890>

Still not matched anywhere else. The bogging-down from ridiculous numbers of Firefox tabs may soon force me to switch, however.

I also like some of the highlight plugins for translation and search.

<comment><USER2> Ah, the tabs tree does look useful.

Opera 10.50 has new search highlighting, though, and the right-click context menu for highlighted text has included a translate option for a while now.

</comment></comment></comment></comment></comment><comment><USER5> I hope they can get some buzz for Opera 10.50. I really love Opera. Try the email client and you know what simplicity means.

</comment><comment><USER6> I tried it. It was pretty fast and rendered everything well, however, without lastpass or 1password support it's tough to fully switch over.

<comment><USER7> Yep 1password is what is keeping me from using it... I used to use opera a long time ago, but was wowed away by 1password and inquisitor on safari...

</comment></comment><comment><USER8> I tried it for OS X, but scrolling sucks and I can't seem to adjust it. There's also some other little things I keep finding that annoy me... It may be faster than Safari, but not compellingly so.

</comment><comment><USER9> I installed it, it lost my saved tabs, and the sidebar looks out of place. I recovered my old version via Time Machine, so until they bring out a stable version, I will wait for thee.</comment>


Concurrency in JavaScript - <AUTHOR> http://typedarray.org/concurrency-in-javascript/

<comment><AUTHOR> i'm waiting for Canvas to be available inside workers. there was a proposal for it a while back [http://lists.w3.org/Archives/Public/public-whatwg- archive/20...](http://lists.w3.org/Archives/Public/public-whatwg- archive/2012Nov/0199.html)

also, a read-only view of a shared array would be massively helpful where the workers only perform analysis without modification. much better than copying or even transferable objects.</comment>


The Battle to Separate Safa and Marwa - <AUTHOR> https://www.bbc.co.uk/news/extra/PLNMqvmycN/conjoined-twins <comment><USER2> Incredibly touching piece, and an astounding display of expertise and skill by the doctors. I found the photo of Rital and Ritaj holding hands while still conjoined particularly moving. Thanks so much for sharing!

</comment><comment><USER3> This is amazing! Great story with good ending.

</comment><comment><USER4> This is unbelievable. Yesterday there was a post well ranked here about the "failure of medicine". I think that besides all well known downsides or modern medicine, we have achieved an amazing progress in the last one hundred years.</comment>


Tardigrades can live 30 years in a freezer&survive in space, and now we know why - <AUTHOR> http://www.theverge.com/2016/9/20/12990274/tardigrade-extreme-water-bears-genome-sequence <comment><USER2> They can splice some of that into my genome.</comment>


StumbleUpon acquires a Montreal startup, 5by - <AUTHOR> http://techcrunch.com/2013/09/24/stumbleupon-acquires-5by/

<comment><AUTHOR> TL;DR

A decade old company (founded in 2001) StumbleUpon has made its first acquisition ever since its inception.

Purchased by eBay in 2007 [1] and then sold to its founders in 2009, StumbleUpon has continued to have stable growth over the past few years barring the lay-offs they had at the beginning of this year. [2]

StumbleUpon earns revenue by using native advertising on their platform called Paid Discovery [3]. Ads are not served through traditional means, rather users will sometimes stumble on a paid stumble and fully engage with that page.

This is also a company that was founded by Garrett Camp, founder of Uber and Blackjet. It seems they have a pretty good sense of direction just like Garrett's other startups.

[1] - [http://investor.ebay.com/releasedetail.cfm?releaseid=246467](http://investor.ebay.com/releasedetail.cfm?releaseid=246467)

[2] - [http://techcrunch.com/2013/01/16/stumbleupon-layoffs- reorgan...](http://techcrunch.com/2013/01/16/stumbleupon-layoffs- reorganization/)

[3] - [http://www.pagemodo.com/blog/driving-traffic-stumbleupon- pai...](http://www.pagemodo.com/blog/driving-traffic-stumbleupon-paid- discovery)</comment>


Announcing TypeScript 4.0 RC - <AUTHOR> https://devblogs.microsoft.com/typescript/announcing-typescript-4-0-rc/ <comment><USER2> Loving the first three changes:

\- Variadic Tuple Types

\- Labeled Tuple Elements

\- Class Property Inference from Constructors

I've been playing with the beta for a pet project for a while and it's been great.</comment>


Russell's Paradox - <AUTHOR> http://www.cs.amherst.edu/~djv/pd/help/Russell.html

<comment><USER2> I prefer the example given in _Logicomix_.

Some books refer to themselves. Most do not.

Suppose I publish a book which is an index of books that do _not_ refer to themselves.

Question: does that index refer to itself?

<comment><USER3> This isn't from Logicomix. Bertrand Russell himself originally sketched out the problem in terms of books and index books. I forget which paper, but it is in the complete collected papers.

<comment><USER2> OK -- the example recreated in _Logicomix_ :)

(Now that you mention it, I think Russell talked about index _cards_ , which wouldn't really make sense to more modern readers)</comment></comment></comment>


30M people in the world are watching video gamers compete for £32,000 - <AUTHOR> http://uk.businessinsider.com/gfinitys-esports-tournaments-draw-30-million-viewers-online-2015-6

<comment><USER2> [http://dota2.prizetrac.kr/international2015](http://dota2.prizetrac.kr/international2015)</comment>


AMD Ryzen Threadripper 3000 32-core CPU is more bad news for Intel - <AUTHOR> https://www.zdnet.com/article/amd-ryzen-threadripper-3000-32-core-cpu-is-more-bad-news-for-intel/ <comment><USER2> Worth noting AMD just announced that the first third-gen Threadripper will come out in Nov with 24 cores, and the planned 16-core mainstream chip (the 3950X) is delayed to November: [https://www.tomshardware.com/news/amd- ryzen-9-3950x-delay-la...](https://www.tomshardware.com/news/amd- ryzen-9-3950x-delay-launch-third-gen-threadripper,40442.html)

A 32-core chip is still almost certain to show up, since benchmarks have leaked, and folks have also leaked some pics of 3950X's and packaging, but I guess supply/demand have kept everything from coming out yet.

As the Tom's post notes, server and client chips use the same chiplets, so it could be that most of the higher-binned ones are going to server parts; some are higher-margin (7742 is almost $100/core, vs. 3950X under $50/core) and plausible there're some enterprise orders (AWS etc.) getting priority. I wonder if they tightened the binning for the 3950X to respond to the fuss over turbo, too.

I had never really considered everything that goes into getting to that SKU/price list and launch date: you don't really know how much supply you'll have at various perf levels, or what demand there will be for what at what price, and if you don't exactly match them you might end up losing money by underpricing, by having to nerf good silicon to fill highly-demanded lower-end SKUs, or by having a shortage that makes customers go elsewhere. (Plus vendors don't just have the public SKU list, they're working out contracts with OEMs/other big customers. And who knows what the competition will do.) High stakes, and practically speaking no backsies; seeing how upset folks are about turbo clocks, imagine if AMD announced a price hike. Glad it's not my job.

<comment><USER3> > As the Tom's post notes, server and client chips use the same chiplets, so > it could be that most of the higher-binned ones are going to server parts; > some are higher-margin (7742 is almost $100/core, vs. 3950X under $50/core)

The 7742 is the "halo" chip, though. Or was, rather, until the Epyc 7H12 was announced.

Other Epycs have much lower $/core. For example the 24-core EPYC 7352 is $1350, making it $56/core. The 16-core EPYC 7282 is actually even cheaper than the 3950X at $650, or $40/core.

No doubt bulk orders are going to get the priority, but the margins may not actually be that different depending on what companies are actually bulk- ordering. The $/core drops pretty quickly even just going down slightly in the stack. The 32c 7452 is $65/core.

And we don't entirely know which aspect of the binning is the limiting factor. If it's just functional cores that's going to be different from if they run at the right frequency/voltage. Epyc is all 225 W or lower. Threadripper 3000 AMD could pretty easily slap a 300W TDP on it and ram voltage through the chips that couldn't cut it at Epyc specs. The 2990WX is, after all, a 250W TDP part. The existing socket is already spec'd for more power capability than the top- end Epyc.

<comment><USER2> I don't really hard-disagree with any of that; we're all guessing. The co. sending chiplets where they fetch the most also doesn't seem outlandish as a factor, though.

FWIW, here's the price-per-core chart for the whole second-gen server line:

[https://www.servethehome.com/wp- content/uploads/2019/09/AMD-...](https://www.servethehome.com/wp- content/uploads/2019/09/AMD-EPYC-7402P-in-7002-Series-Cost-Per-Core.jpg)

(Doesn't factor in voltage/freq needs for different SKUs, some of them needing fully-working chiplets, etc. Still.)

The low pricing on the 7282 and a few others is interesting, too. Wonder if it's a factor that it can use partially-working chiplets since the server I/O die can take up to eight chiplets and the client die only two, or if that's totally unrelated.

</comment></comment></comment><comment><USER4> AMD is in a really good position right now.

Intel still has the lead in low idle power which is good in laptops.

Ryzen lets different cores have different max frequencies so if your code is single threaded and your operating system isn't the newest that could be a reason to go Intel. Likewise if its single threaded and can take advantage of AVX-512 or the Intel Math Kernel Library.

But otherwise?

<comment><USER5> The main selling point for Intel isn't speed anymore.

Intel still has superior performance counters and debugging features. Mozilla's rr (Record and Replay framework) only works on Intel for example, and Intel vTune is a very good tool. AVX512 is also an advantage, as you've noted.

There are other instruction set advantages: I think Intel has faster division / modulus operator, and also has single-clock pext / pdep (useful in chess programs).

For most people however, who might not be using those tools, I'd argue that AMD's offerings are superior.

<comment><USER6> Is AVX512 really an advantage though?

Intel is infamous for severely downclocking the processor for these and other AVX/SSE family instructions, to a point where sometimes using them makes the program slower than it would be otherwise, especially if you're constantly provoking frequency switches between them and regular instructions.

AMD might not have implemented AVX512 specifically yet (there's nothing legally keeping them from doing so however, they have patent sharing agreements with Intel regarding the entire x86/x64 ISA and extensions), but what they currently DO have is all common SIMD extensions implemented (up to SSE4 and AVX2 if I'm not mistaken) without incurring any frequency penalties on clock speeds for using them.

I can live without AVX512 for now, even though I'd be happier to have it. But I would really rather not have it if it came out in the same crap implementation that Intel has.

<comment><USER7> You can look at binning statistics for non-avx/avx2/avx512 clock speeds: [https://siliconlottery.com/pages/statistics](https://siliconlottery.com/pages/statistics)

For example, the worst 7980XEs do 4.1/3.8/3.6 GHz for each of these respectively. 0.5 GHz down clock isn't too bad. You can change these settings in your bios however you'd like on the unlocked CPUs (ie, the HEDT lineup + W3175X). I do find those down clocks are necessary; I've passed 100C with a 360mm radiator and roaring fans. AVX2 loads don't get anywhere near that hot.

But for all that, I do see a substantial benefit on many workloads from avx512 -- at least 50% better performance than what I'd get from avx2.

I definitely think it's nice to have, especially if you enjoy vectorizing code and looking at assembly. With much bigger performance wins on the line, it's more rewarding and more fun -- and you have more tools to play with, like vector masking (unfortunately, gather/scatter have been disappointing). Fun or not though, if you offer me avx512 on one hand vs twice the cores with full avx2 for the same price on the other, I'd have a hard time rationalizing avx512.

<comment> 0-_-0 > I've passed 100C with a 360mm radiator and roaring fans

Damn, which radiator? Was there a GPU under load in your water loop?

<comment><USER7> Celsius S36. My GPU is on a separate loop and was idle at the time.

I was running benchmarks of Intel MKL's zgemm vs zgemm3m because of a Julia PR that recommended replacing the former with the latter. I don't think anything hits a CPU quite as hard as a good BLAS.

I think my thermal paste may be bad, because the CPU idles hot -- nearly 35 C. I ordered a Direct Die-X and MO-RA-420 radiator, so I'm planning on swapping the AIO for an open loop with way more radiator area and flow through the fins.

Running dgemm, that CPU would hit just a tad below 2 teraflops. I'd like to get it just over that (and run much cooler).

</comment></comment></comment></comment></comment></comment><comment><USER8> I just built a new workstation and went AMD. I'm amazed by the performances. I have a threadripper 1920 and it performs really well under many different loads (compiling, gaming, video editing). It also handles my dev vms very well. Of course this is purely based on my feeling, but I had an i7-9700 before and while I had faster FPS in some games, it was really bad at handling vms.

This is subjective, but I have a really good feeling about AMD, both on CPU and GPU side.

Only thing I could ask is for more open source (open source CPU firmware), but I think this is only a dream.

</comment><comment><USER9> I have the original Threadripper 1950X overclocked to 4.1ghz and I've been pretty happy with it. As a developer I can't find much reason to upgrade 2 years after its release. Single core improvements (from AMD or Intel) aren't game-changing. More cores won't do anything for me (.NET Core, angular, etc) I've considered switching back to intel for their 5ghz processors but based on benchmarks, I wouldn't see anything but marginal improvements. All in all, competition is good, but I wish they'd focus on single core performance improvements.

<comment><USER10> They're doing both. AMD has ramped single-core performance, at least from an IPC perspective, massively. From Excavator (2015) to Ryzen we saw a 52% increase in IPC. From Ryzen 1000 to 2000 (Zen+) we saw a 3% increase in IPC. From Ryzen 2000 to 3000 (Zen2) we saw a 15% increase in IPC.

From 2015 to 2019 we saw a total IPC boost on the AMD side of over 80%, and an increase in max core count in the desktop line of 433%, from 6 to 32.

I'm okay with this progress :)

<comment><USER9> The progress is ok, just incremental. AMD had been playing catch up, until Ryzen/Threadripper.

Just for comparison, here are the numbers from my GeekBench 4 run:

Single-Threaded: 4,746 Multi-Threaded: 34,586

According to a leaked benchmark, the Threadripper 3000's numbers are:

Single-Threaded: 5,519 Multi-Threaded: 68,279

The multithreaded benchmark is 2x, that's a no-brainer since it's likely to have 32 cores vs the 1950x's 16 cores. Now, I will say that TR3000 benchmark is not overclocked (3.6ghz.) But from what I've read, it seems like there's not much room for these latest chips to be overclocked. So, despite being the 3rd iteration, the TR3000 is only 16% faster in single-threaded benchmarks than my 1950X.

Intel's 9900K (overclocked) gets a single-threaded GB4 score of ~7,000. That (or its successor) may be my next machine.

<comment><USER11> Honestly, that seems rather low. It looks like a lot of Ryzen 3900X's are doing well over 6000 on Geekbench? Ie. [https://browser.geekbench.com/v4/cpu/14151649](https://browser.geekbench.com/v4/cpu/14151649)

Is the Threadripper 3000 test also from Geekbench V4? I noticed they added V5.

I don't see why it should be so far behind the Ryzen in single thread... unless the boost isn't working properly or something, could be disabled if it's a test chip.

<comment><USER9> Yep, it's from GBv4. It could be that the 3900X is clocked higher than the TR3000. Given that they're packing more cores on the chip, it's possible that they can't dissipate heat as well and limit the single-core top-end speed on the TR3000s more than on the Ryzen 3000 series.

</comment></comment></comment></comment></comment><comment><USER12> But will they actually be generally available for purchase? I was super excited to grab an AMD Ryzen 3900x when they came out mid-July and they've been consistently sold out since then. That's over 3 months with the only reliable way to get one being to buy one for almost twice the MSRP from 3rd party resellers at $800-900 vs MSRP of $500.

<comment><USER13> demand in US is really high. in my country (SEA), 3900x is available almost every big computer shop.

</comment></comment><comment><USER14> Rumor has it there's a 64 core chip coming ~ this year too! With 128 threads, of course.

The following article calls it a "one-two punch"... wow!

[https://www.digitaltrends.com/computing/amd-ryzen- threadripp...](https://www.digitaltrends.com/computing/amd-ryzen- threadripper-3000-cpu-64-core-2019/)

</comment><comment><USER15> I'm ignorant in stock trading, I admit. But I dont understand why AMD stock isn't blowing up. It's done a recovery after a market wide dip, but hasn't all this Intel beating news raised the hype? So many other tech companies are so severly overvalued and hyped daily. Why is AMD a steady 28-32?

<comment><USER16> With a p/e of 166, there's a lot of optimism priced in.

<comment><USER17> Even more amazing when you compare it to Intel with a p/e of 11

</comment></comment></comment><comment><USER18> By what mechanism would the multicore performance rise 2x without increased single-core performance? I'm kind of confused. Eliminating intra-CPU concurrency bottlenecks, I guess?

<comment><USER19> Which benchmark or comparison are you referring to?

<comment><USER18> Pretty much the only numbers in TFA:

> The single-core score of 1,275 is pretty much the same as for the current > flagship Threadripper 2990WX, ...

> But when it comes to multi-core, the Threadripper 3000's score of 23,015 > absolutely destroys the Threadripper 2990WX's score of 13,400, ...

<comment><USER20> 2990WX was a weird ultra-NUMA setup where half the dies had no direct memory access at all. Part of the gain will have been fixing that - particularly on Windows where the scheduler did not understand this configuration at all (Linux results have always been much better).

Another part is the power consumption and resulting higher clocks. The 2990WX was basically power limited, the new chips are on 7nm which will drastically reduce their power consumption and allow higher clocks inside the same power envelope.

<comment><USER18> Wouldn't the higher clocks impact single-core performance? The article claims that's about the same. I suppose the userbenchmark single-core perf test[1] (the article links to some userbenchmark.com results) could just be a bad test; I'm unfamiliar with it. (It could also be that 2990WX wasn't power/heat limited on a single core, and neither is 3xxx; but say, at 16 core 100% clock, 3xxx can go higher than 2990WX due to a lower power process?)

Anyway, it's still 32 cores. While eliminating the ultra weird NUMA effect and increasing clocks could explain a significant benefit, I'm still really struggling to intuit 72% higher performance. But it is Windows 10, so your remarks about the Windows scheduler may explain the gap.[2]

[1]: [https://cpu.userbenchmark.com/Faq/What-is-single-core- intege...](https://cpu.userbenchmark.com/Faq/What-is-single-core-integer- speed/72)

[2]: [https://www.userbenchmark.com/UserRun/19698768](https://www.userbenchmark.com/UserRun/19698768)

<comment><USER20> Zen2 is moving to a somewhat deceptive advertising model for clocks.

AMD is advertising the absolute highest that any core on the die can hit under extremely light load for an absolute instant. Sustained single-core clocks will be 100-200 MHz less and sustained all-core will be significantly less. Most cores on a chip are not capable of sustaining the advertised clockrate even under severe voltage and even for instant loads, only the "preferred core" on a chip. There is a significant "binning effect" not just between chips, but between individual cores on a chip.

Intel and previous generations of AMD cores used to advertise the sustained single-core rate, which could be achieved on any of the cores. This is a changeup to how the clockrate has been advertised.

original: [https://www.youtube.com/watch?v=DgSoZAdk_E8](https://www.youtube.com/watch?v=DgSoZAdk_E8)

followup after a patch: [https://www.youtube.com/watch?v=3LesYlfhv3o](https://www.youtube.com/watch?v=3LesYlfhv3o)

As such the clockrates may be significantly different from what you're intuiting based on the advertising.

<comment><USER18> I don't actually recall any Zen2 clock advertising in particular. My comment was purely in response to your earlier statement:

> Another part is the power consumption and resulting higher clocks. The > 2990WX was basically power limited, the new chips are on 7nm which will > drastically reduce their power consumption and allow higher clocks inside > the same power envelope.

</comment></comment></comment></comment></comment></comment></comment><comment><USER21> But what processor is best for Dwarf Fortress?

<comment><USER5> Dwarf Fortress is IIRC mainly limited by RAM latency. So you really should be trying to get 3600 MHz RAM with as tight timings (maybe CL16 timings?) as possible.

Dwarf Fortress's simulation is all about pointer-indirection and jumping around memory. The CPU doesn't really do much except wait for RAM most of the time. It takes ~50ns to talk to RAM, but the CPU is clocked at 4GHz (0.25 nanoseconds), giving you an idea of scale. The RAM tightening can bring your latency anywhere from 50ns to 200ns depending on how well you tune your RAM parameters, and depending on chips and stuff. (Servers usually have lots of slow LRDIMM RAM over multiple-sockets that can be 200ns latency or worse).

AMD takes their design out of the server-playbook, and seems to have ~100ns main DDR4 RAM Latency. So Dwarf Fortress probably will be faster on Intel i9-9900k (monolithic design with integrated memory controller and 50ns main memory latency).

<comment><USER22> Interesting! Is there a simple way to measure memory latency?

<comment><USER5> Create 1-billion 32-bit integers numbers between 0 to 1-billion. Knuth shuffle the integers.

"Linked list" traverse the integers as follows:

        //array is full of 1-billion numbers, randomly sorted
        uint32_t idx = 0;
        for(int i=0; i<200000000; i++){
            idx = array[idx]; // Random traversal
        }

Pull out a stopwatch (or use Linux's "time" functionality). Divide the time by 200000000. You've now measured memory latency.

<comment><USER23> I suggest using Sattolo' algorithm instead.

Here is a relevant blog post [https://lemire.me/blog/2018/11/13/memory-level- parallelism-i...](https://lemire.me/blog/2018/11/13/memory-level-parallelism- intel-skylake-versus-apple-a12-a12x/)

</comment></comment></comment></comment></comment><comment><USER24> The article references a Geekbench score that's now taken offline.

Google Cache: [http://webcache.googleusercontent.com/search?hl=de&ei=ewaGXa...](http://webcache.googleusercontent.com/search?hl=de&ei=ewaGXa_NDoLSkwWai5LwBg&q=cache%3Ahttps%3A%2F%2Fbrowser.geekbench.com%2Fv5%2Fcpu%2F170821&oq=cache%3Ahttps%3A%2F%2Fbrowser.geekbench.com%2Fv5%2Fcpu%2F170821&gs_l=psy- ab.3...1047.2108..2466...0.0..0.120.464.6j1......0....1j2..gws- wiz.......0j0i67j0i131.ZfZMOTb1Yww&ved=0ahUKEwjv9peD5eHkAhUC6aQKHZqFBG4Q4dUDCAo&uact=5)

Results (also in the article): Geekbench v5.0.1 Tryout for Windows x86 (64-bit); 1275 single core, 23015 multi-core for 32C/64T @ 3.59GHz Base Frequency (and with 32GB DDR4, no info on 4 or 8 channel).

</comment><comment><USER25> As more people move to city cores from rural areas... I think computers are becoming the next generations 'car', in terms of an object to soup up with new parts.

<comment><USER26> Interesting metaphor. It has been this way since the 1980s. For instance, in 1985/86 it was popular to upgrade your 386sx with an i387 math co-processor which is something like adding a cat back exhaust system on your car. Later on increasing RAM, installing QEMM for boosting available memory for DOS games and over clocking and then add on 3D video accelerator cards. There was even a physics add-on card at one point. Mostly these focused on performance gains. The shift towards LED bling and aesthetics is more recent though.

</comment></comment><comment><USER27> Now if only programming multi-core work could catch up to this hardware. Honestly chips with more cores is starting to sound like cars with more wheels or I guess more valves lately.

Doesn't matter the horsepower the speed limit is the same until we can make software better.

<comment><USER28> This is a very outdated complaint, we are using all cores these days, just not on all systems. Backend software is concurrent & distributed, and although utilization is lower than we'd like, the extra cores are not going to waste.

However, if you put this in a machine you use for playing games or checking Facebook then you might think that it's a waste, and you'd be right. There is plenty of software out there which is still single-threaded, but it's not dominating our utilization.

<comment><USER29> Honestly with the way chrome and firefox behave nowadays, even grandma looking at facebook benefits from multiple cores

<comment><USER30> Not really, any window/tab that is not in focus gets heavily throttled by the browser, gradma is probably not running any extensions and single websites are still single threaded for most parts (more and more are adding web workers for various parts though)

<comment><USER31> >not running any extensions and single websites are still single threaded for most parts

Such a shame that all that JS botnet is still single threaded, isn't it ?

</comment></comment></comment></comment></comment><comment><USER32> _> The single-core score of 1,275 is pretty much the same as for the current flagship Threadripper 2990WX, and is actually slightly lower than the 1,334 that the Intel i9-9900K scores._

In what world is this a valid comparison? No one shopping for a $500 8 core consumer cpu is looking at a $1700 32 core hedt chip. They're entirely different markets. If you can use 32 cores you'd be looking at intels x series at a minimum, if not xeons. This is silly at best, if not intentionally misleading.

</comment><comment><USER33> I'd rather choose really big caches per core than really big number of cores.

<comment><USER3> Zen2 is 16MB L3 per ccx across the entire stack (hence why even the Ryzen 5 3600 is 32MB of L3 - it has 2 CCXs). There's no reason to assume this won't be true on Threadripper 3000. So a 32-core Threadripper 3000 would have at least 128MB of L3.

<comment><USER33> L2 cache is what matters. Zen2 has just 512 KiB per core. My old Pentium-M had 2MB. I wish modern CPUs had at least this much.

<comment><USER3> Performance is what matters, not cache size. Your Pentium-M didn't even have L3, hence why its L2 is so big.

</comment></comment></comment></comment><comment><USER34> Why do y'all hate Intel so much?

<comment><USER35> Because they're the dominant player (and the history of getting to that dominance is very sketchy, including $billion fines in the EU for bribing OEMs to use their processors)

</comment></comment><comment><USER36> Why is AMD making these chips with 14nm transistors instead of using TSMC's 7nm stuff? AMD is only catching up to Intel because Intel's 10nm has been delayed so much, but if Intel can ever figure out their 10nm process they'll be in the lead again.

<comment><USER37> AMD use 14nm for the IO die and 7nm for the compute chiplets. 10nm won't help much - after all the security fixes that Intel have had to implement, AMD have the IPC advantage.

<comment><USER10> And further, "nm" these days are marketing numbers. It's totally possible that AMD's 7nm isn't directly comparable to a 7nm (or 10nm) Intel process.</comment></comment></comment>


Ask HN: Inexpensive 800 number solutions like Twilio? - <AUTHOR>

I'm looking at twilio for an 800 number solution currently. I would also consider easy to configure voice mailboxes with 800 numbers.<p>Can anyone here give me their experience with twilio? For people with 800 numbers, do you find it helps with credibility?  If you have any other recommendations for this type of service, I'm all ears. <comment><USER2> In this page you can find Toll free numbers from $2.oo / month and $2.oo setup fee <http://www.toll-free800.com>

<comment><AUTHOR> This looks excellent. Do you use them?

<comment><USER2> I do . I have 2 toll freenumbers there

</comment></comment></comment><comment><USER3> Twilio is sweet. Very easy to use, great documentation, and most importantly a well-designed API.

</comment><comment><USER2> I do, have 2 toll free numbers there</comment>


A compiler for a fully Turing complete language in 20 lines of JavaScript - <AUTHOR> https://github.com/olahol/bf2c/blob/master/index.js <comment><USER2> Flagged for the false title. If you are going to allow require, I will make a fully Turing complete language in 1 lines of JavaScript.

require("bf2c");

</comment><comment><USER3> How cool.

Too bad I don't understand it.

<comment><USER4> You will after reading [https://en.wikipedia.org/wiki/Brainfuck#Commands](https://en.wikipedia.org/wiki/Brainfuck#Commands)</comment></comment>


The Atlantic president explains Scientology advertorial 'screw-up' - <AUTHOR> http://jimromenesko.com/2013/01/19/the-atlantic-president-explains-the-scientology-advertorial-screw-up/

<comment><USER2> > _"We ran a "native advertising" campaign for a new advertiser..."_

We ran ads that look like genuine content...

> _"...that, while properly labeled as Sponsor Content, was in my opinion > inconsistent with the strategy and philosophy for which this program is > intended."_

...and we thought you wouldn't notice, or care, but clearly I was wrong and we look like giant douches.

> _"In this case, we did not adequately work with the advertiser to create a > content program that was in line with our brand."_

It's the _advertisers_ fault.

> _"In addition, because we had not fully thought through the issues around > commenting on Sponsor Content, we made some mistakes trying to moderate the > commenting thread."_

It's the commenter's fault.

> _"The general media climate also played a role here."_

It's the media climate's fault.

> _"Once these issues came to light and I had the opportunity to assess the > campaign, I made the decision to suspend it pending further review."_

When the shit hit the fan, I panicked.

> _"To be clear, our decision to pull the campaign should not be interpreted > as passing judgment on the advertiser as an organization."_

Don't fear, we'll still run ads for anyone...

> _"Where I believe we erred was in the execution of the campaign."_

It's probably our fault.

> _"We then issued a statement to the press admitting we were at fault. When > we make a mistake, we admit it."_

See, I admitted it before. I get kudos for that.

> _"Our highest priority is The Atlantic's reputation and credibility. That's > why so many readers trust us and why advertisers want to work with us."_

We're willing to sell out our reputation and credibility, as long as it doesn't backfire.

> _"Why did it happen? Quite simply, we did not have clearly established > digital advertising guidelines and policies in place..."_

It's management's fault.

> _"...and when you're innovating in a new territory without standardized > guidelines (we're not alone in the industry on this issue, by the way), > mistakes can happen."_

It's "innovation's" fault.

> _"One important note for everyone: casting blame on any group or any > individual is both unfair and simply not what we do at The Atlantic."_

It's not _my_ fault.

> _"And we most certainly should not speak to the press or use social media to > attack our organization or our colleagues. We are a team that rises and > falls together."_

Be quiet about it.

> _"What is our plan going forward? We are currently finalizing new policies > and guidelines to govern advertising overall, with a specific focus on > Sponsor Content."_

We're figuring out just how far we can push when lying to customers.

> _"Very shortly, we'll publish these new policies, and I'll be discussing > them publicly with the press."_

Hopefully this will blow over soon.

> _"My hope is that we'll turn this issue into a moment where, as a leader in > digital advertising, we will help move the industry to a better place."_

If I can spin this, it'll look great on my CV.

> _"If you have ANY questions or potential concerns about something you're > working on, please don't hesitate to push it up the chain. Push it up to > me—and if I think I need to, I will include Justin and David, not to mention > Linda, Natalie, Bruce, Aretae (our new deputy general counsel), etc."_

We're watching our collective arses closely now.

> _"It seems fitting to quote one of our founders, Ralph Waldo Emerson, who > once said "Our greatest glory is not in never failing, but in rising up > every time we fail." This isn't the first, nor the last time that WE will > make mistakes..."_

WE

> _"..., but what is important is how we handle them and what we learn from > these moments."_

I'm not going anywhere.

> _"I am available at any time (including this weekend) should you wish to > discuss any aspects of this week in further depth."_

I'm so contrite, I'm even working this weekend.

<comment><USER3> Well this is a cynical take...

>> _"We ran a "native advertising" campaign for a new advertiser..."_

> _We ran ads that look like genuine content..._

... yet clearly marked as advertising...

>> _"...that, while properly labeled as Sponsor Content, was in my opinion inconsistent with the strategy and philosophy for which this program is intended."_

> _...and we thought you wouldn't notice, or care, but clearly I was wrong and > we look like giant douches._

That's an unfair characterization. Another way of looking at this is that they did, indeed, realise they were wrong and looked terrible, and so realized they had made a mistake and suspended the program.

>> _"In this case, we did not adequately work with the advertiser to create a content program that was in line with our brand."_

> _It's the advertisers fault._

What part of " _we_ did not adequately work with the advertiser" turns this sentence into an accusation against the advertiser?

>> _"In addition, because we had not fully thought through the issues around commenting on Sponsor Content, we made some mistakes trying to moderate the commenting thread."_

> _It's the commenter's fault._

What part of " _we_ had not fully thought through the issues around commenting" and " _we_ made some mistakes" turns this sentence into an accusation against commenters?

>> _"The general media climate also played a role here."_

> _It's the media climate's fault._

Fair comment, this does seem to be a deflection towards the "general media climate". Very unnecessary, it was _The Atlantic_ , and _The Atlantic_ only who solely made this mistake.

>> _"Once these issues came to light and I had the opportunity to assess the campaign, I made the decision to suspend it pending further review."_

> _When the shit hit the fan, I panicked._

Possibly. Or he could have realised that they had made a massive mistake, not implemented very well and did the responsible thing which was to pull down the campaign.

What would you have had him do? Keep the campaign running?

>> _"To be clear, our decision to pull the campaign should not be interpreted as passing judgment on the advertiser as an organization."_

> _Don't fear, we'll still run ads for anyone..._

As he says below, he believes the way that they ran the campaign was at fault here, and had nothing to do with the advertiser. The fact that the advertiser was the Church of Scientology had nothing to do with their decision. Seems like a fair comment to me.

>> _"Where I believe we erred was in the execution of the campaign."_

> _It's probably our fault._

It's sort of funny how above you say that he's blaming: the media climate, the advertisers and the commenters, yet here you see that he is claiming responsibility.

>> _"We then issued a statement to the press admitting we were at fault. When we make a mistake, we admit it."_

> _See, I admitted it before. I get kudos for that._

Well... yes! There are plenty of organizations I've seen who have tried to spin it that it wasn't entirely their fault. He sure does get kudos for admitting fault - that's not easy to do. Many organizations won't do it (witness Carmen Ortiz's non-apology apology).

>> _"Our highest priority is The Atlantic's reputation and credibility. That's why so many readers trust us and why advertisers want to work with us."_

> _We're willing to sell out our reputation and credibility, as long as it > doesn't backfire._

Oh come on! They tried something they hadn't done before to get some advertising dollars, they totally stuffed the execution and now you are saying they are total sell-outs? I think that's a bit much, really.

>> _"Why did it happen? Quite simply, we did not have clearly established digital advertising guidelines and policies in place..."_

> _It's management's fault._

Well... yes! However, he's saying here the specific areas where they erred. I think that's constructive and reasonable. It also shows that they have identified the areas in which they need to improve so the problem doesn't occur again.

Unless they admit fault, and work out where things went wrong, then they can't exactly improve on where they made bad judgement calls and errors.

>> _"...and when you're innovating in a new territory without standardized guidelines (we're not alone in the industry on this issue, by the way), mistakes can happen."_

> _It's "innovation's" fault._

Uh, no. He's saying that this wasn't something they had done before, it was untried and untested and they made a lot of mistakes as they were, simply put, inexperienced.

>> _"One important note for everyone: casting blame on any group or any individual is both unfair and simply not what we do at The Atlantic."_

> _It's not my fault._

That is extremely unfair. I actually thought this was a remarkable ethos, given the number of companies I've seen that are willing to find a scapegoat when things go bad.

There's no point in pointing fingers at any one person at _The Atlantic_ , because it's not constructive. While you may characterize this as insulating Scott Havens from criticism, more likely he is publicly defending the entire team from unnecessary blame.

>> _"And we most certainly should not speak to the press or use social media to attack our organization or our colleagues. We are a team that rises and falls together."_

> _Be quiet about it._

He's saying that as an organization, they aren't going to publicly air their dirty laundry or have insiders leak criticism to attack one another. Fair enough, don't you think? Or do you think that employees should go at each other in the public arena?

>> _"What is our plan going forward? We are currently finalizing new policies and guidelines to govern advertising overall, with a specific focus on Sponsor Content."_

> _We're figuring out just how far we can push when lying to customers._

Or they are trying to work towards tightening their advertising policies to more clearly distinguish between sponsor content and content from _The Atlantic_. In other words, this debacle has caused them to review what went wrong, identify areas that aren't working or are broken, and then to fix those policies and guidelines.

Sounds like a reasonable thing to me!

>> _"Very shortly, we'll publish these new policies, and I'll be discussing them publicly with the press."_

> _Hopefully this will blow over soon._

What? They are going to publicly publish their advertising policies and discuss them fully. That's called "transparency", the direct opposite of "Hopefully this will blow over soon".

>> _"My hope is that we'll turn this issue into a moment where, as a leader in digital advertising, we will help move the industry to a better place."_

> _If I can spin this, it'll look great on my CV._

Given your responses, you seem to be pretty good at twisting the words of others yourself.

>> _"If you have ANY questions or potential concerns about something you're working on, please don't hesitate to push it up the chain. Push it up to me—and if I think I need to, I will include Justin and David, not to mention Linda, Natalie, Bruce, Aretae (our new deputy general counsel), etc."_

> _We're watching our collective arses closely now._

Or he is asking for input from employees and is willing to take on board feedback.

>> _"It seems fitting to quote one of our founders, Ralph Waldo Emerson, who once said "Our greatest glory is not in never failing, but in rising up every time we fail." This isn't the first, nor the last time that WE will make mistakes..."_

> _WE_

Yes, "WE". _The Atlantic's_ ethos appears to be to run as a unified team. I would hazard a guess, but I'd say there were many people who came up with the advertising campaign.

>> _"..., but what is important is how we handle them and what we learn from these moments."_

> _I'm not going anywhere._

I didn't read that at all. He said that when mistakes happen, which they will, they will handle them appropriately and make adjustments as necessary.

>> _"I am available at any time (including this weekend) should you wish to discuss any aspects of this week in further depth."_

> _I'm so contrite, I'm even working this weekend._

This was an internal memo. Yes, he does appear to be working on the weekend and is making himself available to employees who are concerned. That's called "leadership" - you know, the sort of thing you exhibit when you have a crisis, which is what this was.

<comment><USER4> >... yet clearly marked as advertising...

That's bullshit. If something uses the same font and page layout as editorial content, it will always be mistaken for editorial content. That's the whole point of it.

And using weasel words like "sponsor content" is definitely not 'clearly marked as advertisment'.

<comment><USER5> To be fair, it was a giant yellow box that gave an explanation of the words "sponsor content" when you hovered over it. Also the box isn't always yellow or that big, sometimes it's light tan, small and blends into the page; so they at least attempted to make it more noticeable in this instance. Not saying they should have put the ad up, for obvious reasons, but people need to pay attention to what they're reading as well.

[http://www.washingtonpost.com/blogs/erik- wemple/files/2013/0...](http://www.washingtonpost.com/blogs/erik- wemple/files/2013/01/atlantic.jpg)

Pop Up Message: "Sponsor content is created by The Atlantic's Promotions Department in partnership with our advertisers. The Atlantic editorial team is not involved in the creation of this content. Email email@example.com to learn more."

[http://www.poynter.org/wp- content/uploads/2013/01/sponsorcon...](http://www.poynter.org/wp- content/uploads/2013/01/sponsorcontentatlanticscien.jpg)

I wonder if this would have even been an issue if it wasn't an article about Scientology, especially since this style of advertising has been happening on that site for at least the last two years and elsewhere since newspapers have been in existence.

<comment><USER2> When I first visited it, it was a plain white text box at the _end_ of the article.

</comment></comment></comment></comment></comment><comment><USER6> _"In this case, we did not adequately work with the advertiser to create a content program that was in line with our brand."_

It's disturbing to see that The Atlantic believes that it could have reconciled Scientology with its "brand" if it had only worked a bit harder. They're only digging themselves into a deeper hole with this corporate mumbo jumbo.

<comment><USER7> It really is a bizarre mess of vague, meaningless, and confusing statements.

> The general media climate also played a role here.

...what?

> To be clear, our decision to pull the campaign should not be interpreted as > passing judgment on the advertiser as an organization.

We don't actually think there's anything _wrong_ with Scientology.

> Very shortly, we'll publish these new policies, and I'll be discussing them > publicly with the press.

We're sure that everyone will forget about this in a week or two, when we'll quietly publish more platitudes about the incident tucked away in a corner somewhere.

</comment></comment><comment><USER8> Some day, I'm going to get a job as CEO of The Smithsonian. I don't care what salary they pay me, but I'll make damn sure my bonus compensation package is tied to profitability.

Then, for the next two years, I'll sell off the museum's collections to the highest bidders, and then leave, having had the two most profitable years ever.

As far as I can tell, this is what passes for genius in today's corporate environment.

</comment><comment><USER9> > From: Havens, Scott

> Date: January 18, 2013, 6:08:51 PM EST

Remember kids, the best time to post non-explanations or non-apologies is as late as possible on a Friday to minimize anyone actually seeing it or acting on it or anything changing at all, really.

</comment><comment><USER10> Here's the original ad archived: <http://www.webcitation.org/6Do5yGrId>

Edit: Ha, the images from the ad are still in The Atlantic's CDN: [http://cdn.theatlantic.com/static/mt/assets/sponsored/0.%20M...](http://cdn.theatlantic.com/static/mt/assets/sponsored/0.%20MR_DAVID_MISCAVIGE.jpg)

</comment><comment><USER11> This response destroys the Atlantic's reputation: blame dodging, inability to see why Scientology is an inappropriate advertiser, and self-aggrandising behaviour are all on display. What a pathetic magazine.

</comment><comment><USER12> It would be great to see the Atlantic covering the Fishman affidavit, Karin Spaink and xs4all against Scientology in one of their upcoming articles.

The world needs a periodic reminder of what Scientology really is and I think the Atlantic is now morally obliged to be the vehicle for that reminder.

<http://en.wikipedia.org/wiki/Fishman_Affidavit>

</comment><comment><USER13> This somehow prompted me to check out the Scientology website. I don't buy into scientology at all (and am pretty against some of the abuses), but wow, they are _really_ good at marketing.</comment>


Ask HN: I'b building an email analytics plugin/app - <AUTHOR> Would you use: - days since reply from [x] - not marked as done - [x] awaits your reply for [y] days - most emailed person last [x] - least responsive person is [x] - most action items from [x] - most requests from [x]<p>Would this increase your productivity? (assuming email is 90% of your day) <comment><USER2> Fairly heavy email user here, but I (personally) wouldn't find use in those metrics. But I like the general idea. I seem to remember that Eudora used to have some analytics.

I recently kept an Excel chart of my inbox, so I could see over time how close I was getting to Inbox Zero and how many emails a day I'd need to get through. Combining that with intelligence on how many emails I receive per day on average to calculate an Inbox Zero date estimate could be interesting.

</comment><comment><USER3> This could be incredibly useful. I drafted something up about this (and some other similar metrics to blend) a year or so back. I ran by an engineer Ive been freelancing with a few years (Im marketing) and they felt it was too much outside their skillset so it got mothballed.

I definitly feel there is a solid oppurtunity here. Good luck with it.

For your stats I'd keep it simple. I think the best 2 here are;

\- most emailed person

\- most actions / requests from

</comment><comment><USER4> I think whether this can help with my productivity depends on how it can integrate with my existing email provider. If it augments the existing experience by working as a browser extension of some sort, it might.

</comment><comment><USER5> How would it work with different email programs?</comment>