Academic success is either a crapshoot or a scam - <AUTHOR> https://grasshoppermouse.github.io/2017/12/05/academic-success-is-either-a-crapshoot-or-a-scam/ <comment><USER3> I would say that social science is an exception rather than the rule when it comes to reliance on p-values for their results. Many, or most scientific fields don't. For example DNA nanotechnology gets published in Nature a lot by creating concrete proof-of-concepts where the "result" is a picture on an atomic force microscope or something like that. Other examples abound, crystallography, microscopy. So saying that academic success is a crapshoot or a scam in general because of p-hacking is not correct. Within social science it is probably correct.

<comment><USER4> Judging by the replication crisis discussions over past two years, I'd add quite a bit of medicine to that list too. And I don't mean just psychology, which is social science with a medical degree - but also drug research.

All of this, though, shows how _hard_ those disciplines are. Relative to that, hard sciences are easy, because they study precisely defined things pretty closely entangled with laws of physics, and you can't easily bullshit mother nature. But as you move away from physics, complexity rapidly increases, the questions themselves become fuzzier, and the answers less clear.

So IMO, the probability that academic success is crapshoot or scam increases with (Kolmogorov) complexity of the subject domain.

<comment><USER5> It's not quite that medicine is more complex, it's that Medicine is practical and accepts not understanding the details. If drug X works then that's enough we will use it even if we don't understand exactly why it works.

The downside is we end up building simplified models well Y is correlated with bad things so we should treat Y as a bad thing. Often this actually works really well, but also opens the floodgates for all sorts of issues.

That's not to say a less practical approach that simply looked for the details would be more effective. But, it would look very different.

<comment><USER6> > It's not quite that medicine is more complex, it's that Medicine is > practical and accepts not understanding the details. If drug X works then > that's enough we will use it even if we don't understand exactly why it > works.

I'm always baffled by this. I can understand why we should move these drugs into practice. But, if the way you conduct research stops at X works - <USER2> should have a good look at the way you conduct research. I can completely understand short term gains but ignoring long term gains is very unscientific and immature.

<comment><USER7> I'm afflicted by three chronic conditions which have no known causes. All of them are treatable; there are guesses as to why the treatments work, there isn't really a good way to establish a sound reason for them working.

You can't just take a person apart to study them while they're alive, and you can't easily reconfigure them to account for variables either.

</comment></comment></comment></comment></comment><comment><USER8> All of my research so far has been either theory/simulation or descriptive (e.g., this is how much this project did cost). After going to a lecture on someone's empirical research two weeks ago, I found myself asking why I have always been kind of turned off by statistical projects. I came up with two answers:

One: at this state in my career I need to publish a lot. There is kind of a speed limit on how fast you can publish empirical work, but with theory the upper bound on your publication rate is how hard you work and how creative you are.

Two: I feel more honest about my theoretical work. My models and simulations are so obviously contrived that there is no danger of anyone mistaking the map for the territory. By contrast, if you come up with some effect of size X in an empirical paper it might be reported in the media as some kind of golden truth like the charge of an electron.

I think if I spent like 12 or 15 months on an empirical project, I also would be so desperate to get a result out of it as to fudge the result. Even among intuitive relationships, the chance you can honestly and provably identify an interesting relationship is really low. For this reason, I read theory and description from all kinds of researchers, but I tend to only be interested in the empirical work from the best-of-the-best. In my field, I think some of the best work is from Rebecca Diamond at Stanford, and one of her most recent papers had a negative result.

I also really discount job market papers: these people spent years on a single project and their whole future is riding on finding something cool, so the incentive to fudge is just extraordinary. It's kind of like trusting a judge to try a case in which his son is the defendant. No matter what he told me about how fair he'll be, I wouldn't believe him.

<comment><USER9> This rings really true to me. I'm switching to more empirical-ish work now, after tenure when I don't have a knife at my throat.

</comment></comment><comment><USER10> One possible, very sad long-term effect could be that the Sciences (results, theories) will be deemed less relevant, less trustworth, less worthy of attention. Eg. I started a Phd in Physics, but I never finished it, I got disillusioned with Academia and some specifics in today's Physics research (like String Theory, Inflation). 5 years later, people are still beating on those drums, and whenever I read about some crazy new model, I think two thoughts: (i) nonsense (ii) I'm so happy I'm not spending my time on this.

<comment><USER11> It would be a very good outcome if society stopped believing anything that comes out of the mouth of a "scientist".

It's just sophistry all the way down.

<comment><USER12> So whose mouths should we trust and why? (Apart from yours)

<comment><USER11> None.

</comment></comment></comment></comment><comment><USER13> > In my field of anthropology, the minimum acceptable number of pubs per year > for a researcher with aspirations for tenure and promotion is about three. > This means that, each year, I must discover three important new things about > the world.

This is typically not the case. Academics will have a number of students working for them and many academics push them incredibly hard while doing very little of the actual research themselves. Instead they spend their time chasing funding. Projects are often built on prior work so many new papers explore new areas of a previous discovery or measure said discovery in a new way. So you're not discovering three new things every year. Some academics will even be bold enough to publish the same data in multiple journals with minor changes in analysis.

With academic research it's often a case of two steps forward one step back. You spend years working towards deep expertise in a field through postgraduate studies and then a postdoc or two. Then when you become academic staff you suddenly become an inexperienced manager and teacher. So you pass the work to inexperienced students who spend years relearning everything and often suffering from over aggressive supervision or complete abandonment. The funding bodies, as is their right, are increasingly looking for commercial outcomes to research so the scope moves from Research to r&D with a little 'r'. So we see research on battery technology making ludicrous claims or graphene in all the things rather than real progress.

This stunts the potential of scientific discovery in favour of pushing revenue generating students through universities and chasing funding for short term gain.

<comment><USER14> > This is typically not the case.

What you're describing is research in fields with lots of grant funding. In some fields, you don't spend your time writing grants, and you don't hire grad students to do your research for you. In those fields, teaching is important because you do a lot of it. In grant-heavy fields, it wouldn't matter if the university shut down their undergraduate programs, because the emphasis would still be on bringing in grant dollars.

<comment><USER15> It may be true that some University/College are interested in research and teaching more than the grant money that is brought in to the department, but there's definitely a bias there. Even more important, if you don't have several graduate students working for you during tenure (supplementing your research & teaching efforts), it's going to be hard to tenure there either. How many classes/labs can you teach without an RA/TA? Remember the big intro courses that new untenured profs get, and that teaching grads is a big part of the job! You basically have to prove you can't handle grad students in some horribly catastrophic way not to get them from the dean/provost.

Perhaps in a field like philosophy or math at a very small school it's possible with amazing individual results, but it's uncommon in my experiences which include watching numerous friends at many different schools (>10) and fields (bio, chem, phys, geo, cs, math, philosophy) both public and private.

Maybe you mean outside the US? I'd believe that. Europe hasn't gone as far down the path of non-tenured teaching faculty as the US, but due to money they seem to be starting to.

<comment><USER16> "if you don't have several graduate students working for you"

And that feeds the birth/death demographics problem where the reason why its hard to get a tenure position is the previous generation had to produce 10, 20, 30 qualified grads in order for them to get their previous generation tenure position, so 10, 20, 30 minus one person are going to get training and education for a position they will never be allowed to fill. I'm sure they'll find something to do with their lives, although to get position XYZ they never trained for, they'll be competing with people who actually trained and studied for XYZ work, and in the end it just means a different name is on the unemployment lists.

Academia is a weird pyramid scheme of human suffering.

</comment></comment></comment></comment><comment><USER17> This article is about "empirical research" which means articles with titles like "Study suggests XYZ".

I avoid these type of "research" articles like plague. There probably exists a "study" that proves and disproves virtually anything. For example, study suggests coffee prevents cancer and vice versa. Such pseudo (and often badly done) statistical voting type of work shouldn't even be called "research" and people doing these shouldn't be called "scientists" (the word "surveyors" seems more appropriate). There is now a cottage industry of pushing out absurd correlations that would be immediately be picked up by media because, you know, so called "scientists" did that.

<comment><USER18> I would be interested in your point if you were revealing something concrete about the OP and their research, but you're just making handwavy claims about broad correlations ("empirical research" = "study suggests xyz")

The fact that you're making overly broad claims accusing people of making overly broad claims doesn't instill confidence that you know what you're saying. If you're such a champion of science, why not state your claims minimally concrete terms. And ideally make falsifiable claims.

<comment><USER17> OP's point is that its hard to get funds to conduct 9 studies per year with hope that 3 will result in paper as expected from the tenure track. I neither have intention to counter argument nor I have insights in to specific field that author is working on (which seems to be anthropology). My argument is exactly more broader as you have mentioned: Lot of research - at least as captured by media - involves simply doing some statistical survey and getting comfortable p-value to suggest some weird correlation. There is a lot of money spent in this type of "research" because it is expensive to conduct such studies. My objection purely on calling this "science" and calling people doing this as "scientists" because these two fairly reputable words are being used to lure general public to believe such "studies". I feel such works should be called "surveys" and people doing it should be called "surveyors". The real scientific work should involve taking the survey only as a hint and then uncover actual mechanism that falsifiably proves or disproves the correlation with hopefully more predictions. Simple surveys don't have these properties and hence such studies shouldn't be labeled as work of scientists.

</comment></comment></comment><comment><USER19> Years ago after spending uncountable beers listening to the frustration of several PhDs friends, I had the idea of a journal for research failures/dead- ends.

The idea was to look at unfruitful research as something useful for the community and give some credit to time and resources spent doing something that did not work (and should not repeated by others). I still wonder if that would create the wrong incentive though and if that could be feasible at all (who likes to talk about money and time wasted?)

EDIT: fixed wrong sentence

<comment><USER20> > I had the idea of a journal for research failures/dead-ends

A bit like the Journal of Negative Results in Biomedicine, the Journal of Negative Results — Ecology and Evolutionary Biology and the psychology Journal of Articles in Support of the Null Hypothesis?

> several phd undergraduate friends

People doing PhDs who are simultaneously undergraduates?

<comment><USER21> Don't forget plant science!

[http://www.journals.elsevier.com/new-negatives-in-plant- scie...](http://www.journals.elsevier.com/new-negatives-in-plant-science)

</comment></comment></comment><comment><USER22> It's the same with everything else. For example, with complex software projects, the ones that make the most incredible claims about their capabilities are much more likely to get hyped up and get blogged/tweeted about... And it's often at the expense of real quality.

<comment><USER23> Methodology merchants like "Uncle Bob" have built careers and even companies on that strategy

<comment><USER12> What 'incredible claims about his capabilities' did he make? (I came in late.)

<comment><USER24> Wasn't he involved in Chrysler's C3 project, the major success of which was the agile programming industry?

<comment><USER23> I always get Bob Martin and Martin Fowler mixed up

</comment></comment></comment></comment></comment><comment><USER25> A bit off the topic but I always wondered why social scientists need funding or even a university for their experiments. I mean the kind where they put 20 people in a room and show them some pictures and ask them some questions. What resources do you need?:

Time - do it as a hobby so it's limited to people who are passionate about discovering something, not just trying to rack up points.

Access to human subjects - get people off the street. Universities can make access to student subjects easier but they're not representative of the world's people anyway so it's not that valuable.

Money to pay the human subjects - save up? People who're passionate about cars, for example, somehow find the money to build modified cars and don't need funding.

Ability - With the oversupply of researchers, maybe just leave it to those who can do it under their own steam? Hopefully undergrad degrees taught them enough to get started. This article says that university researchers aren't doing it with skill anyway but with luck or cheating.

Motivation - if nobody can be bothered investigating some effect, then maybe it's not really that interesting anyway and there's not much value in studying it?

What am I missing?

<comment><USER26> Depends what you study. If you're an economist you need access to data like compustat or other. If you're a political scientist in the US much of the data is available but you also need to spend a while cleaning it (although there are great resources now too.)

You're really talking about small scale interviews or qualitative work. That's an aspect of social science but it's not all of it.

</comment></comment><comment><USER27> > empirical social scientists with lots of pubs in prestigious journals are > either very lucky, or they are p-hacking.

That's the money quote. This is mainly applicable to "empirical social science", which many already know as a big scam.

<comment><USER26> Absolutely. Been there, done that. I worked for a guy who is top in his social science field. Total p-hacking was the norm. It was an eye-opening experience for someone new to the field expecting things to be done more honestly following something like the scientific method.

Once I found a major bug in some code that was the basis of a significant chunk of his major book and an important article. (Basically an optimization process that didn't work but gave some nice results he built a story around.) When I wrote him a report on what was wrong and why, he didn't respond. I was supposed to write a paper based on the same code and all he would say was along the lines of "it should work". Over time I realized that the implication was that I should "make" it work. Of course, he never said those words but over time the message was clear. After beating my head against a wall for a long time, I eventually walked away from the project. Technically bad for my career, but I honestly couldn't see the point in being at a top university essentially selling snake oil.

That being said, there are fields of social science that are much better: demography, political science, economics and even psychology (which is very difficult at the best of times!) are trying to clean up their act. Basically, any field of study with consistent data available openly is probably pretty decent.

There are other fields like business/management where the pay is generally good (i.e. motivation for a nice job with good perks is there), many of these journals don't require data to be released, and the samples being studied tend to be convenience samples or impossible to reproduce. I generally don't trust the work in this area.

Sadly, policy makers are often willing to take questionable academic studies that support their preconceived notions. When these things don't work out, the general public becomes jaded that their taxes were wasted... it makes all social science research look bad.

</comment></comment><comment><USER28> One of the things I find surprising about the way we do science today, is the idea that "data is king" and that it "speaks for itself". In reality, data devoid of context, is absolutely meaningless. If the goal of scientific researchers and journals is to promote good science, it's absolutely shocking that practices such as pre-registration of hypothesis and methodologies, are still completely optional.

[https://wp.me/p3SyBR-8t](https://wp.me/p3SyBR-8t)

<comment><USER29> I'd be surprised if anyone is really arguing that the context of data is not important.

<comment><USER28> And yet, practices like pre-registration of hypotheses are still not widespread. If they did, it would eliminate so much of the cherry-picking of data that occurs.

[https://xkcd.com/882/](https://xkcd.com/882/)

</comment></comment></comment><comment><USER11> If everybody is required to publish tons of stuff, you're just creating tons of stupid stuff no one will read, and that crap is going to cover the really important stuff.

</comment><comment><USER30> One has to wonder whether any of the qualitative social sciences really even have a right to soundblast about concepts that have no provable metric...no way to be tested, and cause no difference in the world whether they are right or wrong. Is it time for the social sciences to throw in the ringer and reform their teaching and relevant arguments into a more mathematical/axiomatic foothold? I for one think so.

<comment><USER31> Isn't the problem trying to force social science into a mathematical/axiomatical model in the first place, when it really is historical and descriptive?

<comment><USER32> I think the issue is more that the social sciences themselves are attempting to operate as the natural sciences do with predictive, falsifiable, and independently actionable conclusions.

History is invaluable and the lessons learned can provide some degree of predictive power. Yet it would be unreasonable to try to argue that something ought be enacted as, as a matter of fact (or at least heavily supported hypothesis), because of results derived from the study of history.

The social sciences in general seem to be want to be allowed to behave as a "historical and descriptive" field, yet be received with the same credulity as, for instance, experimental physics. Ultimately, I think the field needs to choose to go one way or the other.

<comment><USER33> The problem is activist social "scientists" making broad claims with no dvidence,and the march to enact sweeping societal changes with their "results".

See: Stereotype Threat, Implicit Association Test and Implicit Bias, Power Posing, et al.

</comment></comment></comment></comment><comment><USER34> I thought quality of journal and number of citations (from quality journals) was more important than pubs per year.

</comment><comment><USER11> If you're going to reward only high-quality studies you'll lose the larger part of scientists, because there are very little of them who can produce anything high-quality.

What you're proposing is mass unemployment.

<comment><USER35> If the studies they are producing aren't high-quality, is there any value to them? Do we, as a whole (because some individuals will lose out, as you say), benefit by rewarding low standards of work?

I'd also said that many may be _capable_ of high-quality work, but merely have limited incentives to produce it currently.

</comment></comment><comment><USER36> I was once asked to edit a sociology paper for grammar, spelling and style, but ended up deleting large swaths of it containing claims made up of thin air and found nowhere in the data or the citations. Of course this was not well received and I expect every one of those passages were promptly restored, complete with their horrendous spelling and grammar.

The claim? That the biggest challenge facing remote poverty- and alcoholism- stricken communities of indigenous peoples were not enough tablets to be able to "Netflix and chill". Because surely that will solve all the world's problems.

<comment><USER37> Would be good if you could substantiate your comment by linking us to the published version.

<comment><USER36> If you don't mind I'd rather not have my name linked with fraudulent academic articles, regardless of who exactly the guilty party is.

I could have just said "What the linked article says is correct", but I figured this anecdote would add more value. But yeah, whether you want to believe it or not, the linked article is pretty accurate. Successful academics either get lucky or they are frauds.</comment></comment></comment>


D.C. accidentally uploads private data of 12,000 students - <AUTHOR> https://www.washingtonpost.com/local/education/dc-accidentally-uploads-private-information-of-12000-students/2016/02/11/7618c698-d0ff-11e5-abc9-ea152f0b9561_story.html <comment><USER2>
        > determined that one person downloaded the document from
        > the website. That person was part of a community
        > organization that has verbally agreed to delete the
        > document, Peabody said.

Fortunate that they were able to do that; considerably softens the blow.

</comment><comment><USER3> If any of the DC public school people want help protecting their student records, I will donate pro-bono hours.

There's one technique that is powerful and rare, which is translucent data access. It's ideal for identifying memberships and trends, such as "how many students are in a class each year" while protecting personally identifying information. A simple common example is a one-way hash function that lets an app confirm a password without needing to store the password in plain text.

<comment><USER4> Send me a ping if you're interested with helping schools with their data - me[at]smaili.org

<comment><USER5> How exactly are you helping schools and their data?

</comment></comment></comment><comment><USER6> Feel there is really no way to prevent this. This was doubtfully uploaded knowingly (or possibly with ignorance). Data dumps occur all the time. As a file, they can too easily be shared. For sure, the school should work on increasing their awareness of handling secure data. But, in the end, nothing would really prevent this from happing again.

<comment><USER7> We do SSL and content inspection of uploads to unexcepted sites to prevent this sort of thing where I work.

<comment><USER8> I'm completely ignorant here, but how do you do that without this? [https://news.ycombinator.com/item?id=11042353](https://news.ycombinator.com/item?id=11042353)

Hoping to learn something, honest question.

<comment><USER9> I'm imagining IT has proviosned certificates on the computers under their control that allows them to do a MITM attack on either blacklisted sites, or non-white listed sites.

</comment></comment></comment></comment><comment><USER10> In my 2nd year of university the chair of the CS department shared an excel file via email containing the private data of all the CS students. ID numbers, addresses, phone numbers, first/last names.

He claimed it was meant for an office assistant but somehow he blasted it to both the BS and MS student lists. I doubt he got more than a slap on the wrist as he remained the chair for another 2-3 years.

<comment><USER11> At my university, a publicly accessible file on the university systems listed addresses, phone numbers, and names for all students, not just those of a particular department. Leaking the same information in an email would have been superfluous. Are those supposed to be secret?

<comment><USER10> I don't know, maybe they are publicly accessible with FERPA. However, considering that some secure systems on (my) campus are pre-registered to our names using just firstname.lastname@example.com with the password being the ID, or some other simple variation thereof, I don't feel very comfortable about it. As much as I wish I lived in a perfect world, I don't, and so it wouldn't surprise me that students don't think to change their original passwords to secure ones.

</comment></comment></comment><comment><USER12> > "Our legal department is now in touch with them to sign off legally that > they will delete the file," she said.

You'd have to be a fool to sign a document. This will just make you liable if the document does get out.</comment>


Tom Loverro's rebuttal to Fred Wilson's post on Marketing - <AUTHOR> http://www.tomloverro.com/2011/02/25/do-startups-need-marketing

<comment><USER2> Either the author didn't bother to read Fred's article or he totally misunderstood Fred's point. All of his counterexamples fully support Fred's point.

A more accurate position for the author to take would be "I diagree with Fred's use of the umbrella term marketing and think he should have said advertising instead."</comment>


Foursquare Introduces New Tools for Businesses - <AUTHOR> http://bits.blogs.nytimes.com/2010/03/09/foursquare-introduces-new-tools-for-businesses/ <comment><USER2> These are great tools. I am particularly interested in how affective check-ins are at bringing in new customers. Foursquare has the data to show who the real social influencers are. Some users may not help, but some could bring dozens of friends with one check-in.</comment>


Show HN: Easter++ AR Game - <AUTHOR> https://apps.apple.com/us/app/headpop-by-face-in-the-hole/id1444809835 <comment><AUTHOR> I am excited to share a (kind of new style of) game that I've been working on: HeadPop!

It's decidedly in contrast with the solo, à la mode, sink-yourself-into-your- phone kind of usage we sometimes see. Ideally, this game gets you up, moving, and interacting with real friends, real time.

You play the game using only your face, and you can play against a partner or a bot. Along the way, you do silly little things, and in the end it outputs a silly little video of you doing the silly little things.

Sometimes life can feel really serious, but hopefully this can bring a little bit of that child-like joy and laughter that we sometimes don't get enough of.

It's pretty simple now, but depending on the reception I may put some more time into developing it! Would love to get some constructive feedback. But more importantly...

Use your head: have fun!</comment>


Utility for faster directory navigation in command-line - <AUTHOR> https://bitbucket.org/amarsahinovic/gb

<comment><AUTHOR> Github mirror: <https://github.com/amarsahinovic/gb></comment>


The Texas Sharpshooter Fallacy - <AUTHOR> http://youarenotsosmart.com/2010/09/11/the-texas-sharpshooter-fallacy/ <comment><USER2> What this comes down to, really, is that people are bad at statistics. The birthday "paradox" is a good example of this; another I like is that in a random bridge hand, there's a ~67% chance that you'll have three cards of the same value and a ~71% chance that you and your partner together will have all four cards of some value.</comment>


The dirty war over diversity inside Google - <AUTHOR> https://www.wired.com/story/the-dirty-war-over-diversity-inside-google/amp <comment><USER2> This was posted already

[https://news.ycombinator.com/item?id=16239253](https://news.ycombinator.com/item?id=16239253)</comment>


Show HN: Shoppy – a better Shopify and Slack app - <AUTHOR> https://shoppy.emerald.io/ <comment><AUTHOR> TL;DR - I just finished developing my first Shopify app--Shoppy, a free app that gives you updates about orders sent directly to Slack and you can even upgrade to Elite and get daily, weekly, or monthly order summaries to make reporting easier. Get the free beta app here: [https://shoppy.emerald.io](https://shoppy.emerald.io). Since this is my first app, any feedback you have is appreciated!

\---

Long version:

Hi everyone -

I've been using Shopify for 2 years as a marketer and ecommerce manager, and I've sold >$500k worth of product in 2018. But, we use Slack, and I've always wanted a Shopify app that sent order notifications for new, refunded, and fulfilled orders to Slack but I also wanted this app to include discount codes used, tags, and UTM tracking for the order so I know what's working and what's not. I tried every app I could find, and nothing had this level of tracking in their Slack notifications.

So, I decided to learn to code and make an app myself.

Shoppy connects your Shopify store with Slack so you can receive important order notifications and summaries sent straight to Slack, reducing the amount of time you spend pulling order data and giving you more time to focus on growing your store. And, because I wanted to solve my own problem, Shoppy is the only app that includes discount codes used, tags, and Google Analytics/UTM associated with our orders. You can also receive daily, weekly, and monthly summaries that include order counts and revenue for that date range, and I'm working on automating these summaries for a future update.

For those curious, the app is built with Rails, hosted on Heroku, and heavily uses the Shopify api gem. It was surprisingly easy to get started, and all the payments are handled by Shopify so I didn't need to bring in a third party payment processor like Stripe.

I just submitted the app to Shopify for review, but I'm looking for beta users to try Shoppy out and provide feedback. I've also included 7-day trials for the Premium and Elite tiers so you can try them out completely for free and you can cancel any time.

If you're interested, here's a link to install the app: [https://shoppy.emerald.io](https://shoppy.emerald.io)

And, if you have any questions/feedback about Shoppy or if I can help answer any marketing or ecommerce questions about your Shopify store, please comment below!

Thanks!</comment>


Will Rails be the new PHP? - <AUTHOR> http://roy.io/jzo <comment><USER2> It's the strangest thing. It's like having a blog and access to Hacker News makes self-proclaimed "seasoned" programers forget that they, too, probably had really crappy code when they started programming.

Instead of bashing green programmers who coincidentally became interested programming at the same time that Rails makes it so easy to start, why not offer them up a list of resources to make them better?

Humans are only born knowing how to cry and swallow. Nobody's born knowing how to program. Let's give Rails the props it deserves for lowering the barrier to entry for making web applications and look forward to the future where these new programmers are making things to make our lives easier.

<comment><USER3> He's not bashing green programmers. He's bashing on programmers with years under their belt who never got any better, and don't participate in any community.

<comment><USER4> > _"I'm worried about the portion of Rails developers who sucked at their > previous language. Or the ones who got started with Rails as their first > "language" without finding the proper (or current) resources or guidance"_

No, he really isn't. He is _totally_ bashing green programmers.

</comment></comment></comment><comment><USER5> Will linkbait be the new inflammatory headline?

Will handwaving be the new fluff?

Will insubstantial murmurings be the new mumbling into whiskers?

Will this turn out to be a useful blog post or is it just a bunch of questions that can't be answered because they're pretty close to meaningless?

</comment><comment><USER6> um no?

PHP "has a relatively small community, with the majority of people working under the radar and on their own?" ?

Where is he getting these stats? relative to what? For instance

[http://www.tiobe.com/index.php/content/paperinfo/tpci/index....](http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html)

Says PHP the language is more popular than Ruby the language (let alone the subset of Ruby programmers who use Rails). I don't get where that perceived smallness comes from. In all my experience and every way I've seen the PHP community is vastly larger than the Ruby community. Also comparing a language to a Framework seems... weird. It might make more sense to compare Rails to Drupal at least, which again, in the enterprise world, I have seen waaay more Drupal installs than Rails ones. There are a few high profile Rails sites but it quickly seems to drop off into hobby and startup. Meanwhile Drupal is just _everywhere_. It's surprising.

Also, accusing the PHP community of "writing below par software while charging their clients for it?" is hilarious coming from the Rails camp. Generally lowering the bar for entrance while making it easier for good devs to get good work done faster also allows more "bad" devs to enter as well. Anyone remember earlier this year that core Rails bug that allowed people to overwrite almost all data that most big Rails sites were affected by (including github)

<https://github.com/rails/rails/issues/5228>

I cannot understand where this viewpoint is coming from, it seems hilarious (and at best, the criticisms seem like the pot calling the kettle black).

<comment><USER7> Sort of a tangent here but I've been trying to decide if our organization needs to move away from using Drupal for custom application development. You see to have some experience, what are your thoughts on it for highly customized web applications?

<comment><USER8> Personally, I've been doing Drupal for a few years now and found the knowledge and skills to be a lot harder-won than the knowledge and skills that I've picked up with Rails in the last few months. The knowledge and skills that I've picked up working with Drupal (HTTP, Databases, etc) more or less translate into general app development, so that surely has something to do with it.

I said this to my wife this morning - "you know what, building stuff in Rails just feels like it accesses my creative brain more than anything I've done before." Having been exposed to some of what Acquia is doing with their cloud hosting stuff I can say with a fair amount of certainty that almost anything under the sun can be accomplished with Drupal, but that's (the creative bit) probably why I'm on a Rails kick lately. It's just more fun (to me). The overhead does feel lower, too.

Sorry if that didn't lend any insight to your actual question.

</comment></comment></comment><comment><USER9> While new developers are coming up learning Ruby as their first language and in the context of Rails, keep in mind that Rails introduces you to much of the core Ruby language, idioms and at least a few design patterns and useful abstractions. It also adds Sass and Coffeescript. There's much more complexity and therefore knowledge required for a "whiz kid" to build a useful Rails app that it's hardly a fair comparison. This "subpar software" meets a need that seasoned developers can't or won't.

Also of note: it's no longer 1999. People expect a bit more out of their website than during the glory days of the independent 19-year-old PHP developer. These novices are orders of magnitude more capable than PHP developers of a decade ago. These folks building one-off websites for clients are also often doing both design and development—something many developers are awful at.

This elitism should stop. Programming is a means to an end, and we should welcome any and everyone despite their level of commitment, talent, or focus.

</comment><comment><USER10> Does Tiger Woods sit around thinking about how all of the weekend hackers at the local country club are making the game of golf worse?

Nope, its the exact opposite, the more popular the game is the more the people at the top of it stand out. If you think you are the shining example of the elite Ruby programmer you should be doing everything you can to get more people involved in the ecosystem, the bigger that gets the better opportunities you'll have in the future.

</comment><comment><USER11> There's really only one paragraph here that remotely tries to explain the headline:

"The PHP where there are lots of people writing below par software while charging their clients for it? The PHP that has a ton of websites with crappy advice and crappy code to copy/paste? The PHP which has a relatively small community, with the majority of people working under the radar and on their own?"

I'm a Python developer. Do you have any idea how much crappy Python code and crappy advice is on the internet? Take a look at the StackOverflow homepage at any given moment. Every language has developers at every stage of the learning process. And every language has developers who have stunted in the learning process.

So, this is ridiculous. The answer is no. Rails provides infinitely better tools in a better language than PHP.

<comment><USER12> > So, this is ridiculous. The answer is no. Rails provides infinitely better > tools in a better language than PHP.

You're making a highly subjective and unqualified comparison of a language to a framework, and (given past citations in similar discussions) probably using information to do so that's at least five years out of date. This after saying you're a python dev so presumably not especially well versed in either language/framework (do correct me if I'm wrong)

Like you say, there's a lot of bad code out there. PHP is notorious for it due to its shallow learning curve and widespread availability on cheap servers. The authors point is that the barrier to entry on ruby/rails appears to be quickly dropping and asks whether it will suffer the same fate.

</comment></comment><comment><USER13> The level of elitism in the Ruby and by extension Rails communities never ceases to amaze me. Having attended several ruby and rails conferences over the last few years, the belief that Ruby is the language of "good programmers" is so prevalent as to be off putting.

<comment><USER14> They aren't mere programmers, they're "rockstar ninjas" (in their own minds).

</comment></comment><comment><USER15> I really enjoy when arrogant geeks tries to bash language X. They will never learn their language is just a tool and as long as you follow best practices it doesn't matter what language you are using.

And i really don't understand why particularly rails developers looks down to newcomers. If you want to help your community just write tutorials, share your experiences from real life, help to newbies and make other's life easier. If you want to entertain other peopler, well , just write crappy blog posts with such title like " Will X be new Y ? "[1]

[1] : But please read first the article about "Betteridge's Law of Headlines" on Wikipedia.

</comment><comment><USER16> Usage of server-side programming languages for websites (30 September 2012):

      PHP      78.1%
      ASP.NET  21.0%
      Java     4.0%
      ColdFusion  1.1%
      Perl     0.9%
      Ruby     0.5%
      Python   0.2%

[http://w3techs.com/technologies/overview/programming_languag...](http://w3techs.com/technologies/overview/programming_language/all)

<comment><USER17> I'm not sure what your point is to begin with, but I am highly skeptical of the scientific nature of this report.

<comment><USER16> "See technologies overview for explanations on the methodologies used in the surveys."

<comment><USER17> I saw it, it's totally hand wavey.

</comment></comment></comment></comment><comment><USER18> This not in my backyard eliteism is a problematic mindset, especially geeks hating on geeks, who probably weren't that popular in school themselves.

Kind of ironic.

Tools are just tools. Most are capable and have good web frameworks.

Opinion and preference is largely what most posts like these are based on, while it's free to have your opinion and preference of what tools you use, judging others, and how and if they may learn is another thing.

</comment><comment><USER19> If you want to compare rails to something compare it to Yii.

If you want to compare something to PHP compare it to Ruby.

Comparing Rails and PHP is just plain dumb.

</comment><comment><USER20> Maybe. If Ruby / Ruby on Rails is a good language for web-development, easy for beginners to learn, the community is helpful, beginners feel welcome, and experienced developers invest in teaching; you'd expect to find a small army of 'just get it to work' developers knocking out small projects and charging too little.

I'm ok with that. Small projects get done. Not everyone needs a $125/hr consultant. Some of those projects will grow and need more experienced developers. All of those beginner developers will gain experience. Some of those beginner developers will continue to grow, learn testing, learn better software design principles and start contributing to the community.

If Ruby on Rails has, or were to have, a small army of beginner developers writing poor code on small projects and charging too little for it; I think it would be a healthy thing.

Disclosure: I got started as a Rails-first-language developer and have loved every second of it. And I try to get new people started with web-development using Rails and hope to see them get their first small projects that they can write bad code for and get paid anything for.

<comment><USER21> > If Ruby on Rails has, or were to have, a small army of beginner developers > writing poor code on small projects and charging too little for it; I think > it would be a healthy thing.

Hmm, well I see your point, but disagree. It's just like the WordPress situation for small businesses.

Someone comes in (usually a "designer") and cobbles some WP plugins and a template into a seemingly okay site for $500, but eventually the small business owner hits the limitations of that arrangement, and (from what I've observed) good programmers won't even take $5,000 to salvage a mess like that.

And not only does the customer end up feeling burned, it also leaves behind some toxic consequences, e.g., the customer has no idea what programming costs nor the difference between a prototype and production quality. So while we all start somewhere, this kind of low-rent ecosystem has certain enduring negative consequences (evident on Craigslist).

</comment></comment><comment><USER22> Taking just the title, not likely. The lesson of PHP is that it is easy to get something running for a beginning programmer, and (probably more importantly) it is very simple for a system administrator to setup. I get the feeling that the first web framework that spends and equal amount of time figuring out how to make system installs and administration easy will take over for PHP.

</comment><comment><USER23> Rails has a steeper learning curve than PHP. And I can't see Rails (easily) being sloppily-disorganized like PHP can.

So I'm going to go with no.

<comment><USER24> You can program messy Ruby as easily as messy PHP. Rails is an MVC framework. PHP has plenty of those as well.

<comment><USER23> Right, but Rails will always have some sort of organization. That being said it _is_ possible to go against the grain and not use Rails' default directory structure - but that's difficult.

I'm comparing Rails to vanilla PHP - just as the author is. People are exposed to PHP's low barrier of entry (again, vanilla here) because you can write an application with PHP/Apache/mySQL very quickly. However, because of this, most instances are very hacky.

<comment><USER15> Please don't compare a framework with a language, even someone before did it.

<comment><USER23> I'm comparing RoR to the act of running PHP scripts on Apache. Both are processes - this is also what the article addresses.

<comment><USER24> You don't think Ruby on Rails requires the overhead of a server???

There are perfectly good and very sophisticated MVC frameworks available for PHP, such as CakePHP (one of the closer to Rails, even down to the command- line "baking" of models, controllers, etc.).

There are meaningful differences to development using both languages, but criticizing PHP because it was popularized independently of any specific framework is not one of them. Especially since there are plenty of situations (think dumb but scalable web services) in which the last thing you want to do is invoke the overhead of a framework in any language.

</comment></comment></comment></comment></comment></comment><comment><USER25> Do some people who do some stuff do stuff better than other people who do stuff worse?

Yes.

</comment><comment><USER26> tl;dr - the author plans to switch to node

</comment><comment><USER27> Where's my $4/month rails hosting?

<comment><USER28> <https://www.appfog.com/products/appfog/pricing/>

</comment></comment><comment><USER29> I've seen this playing out for at least five years.

For a long time rails has attracted developers who are strong in design but weak in programming.

</comment><comment><USER30> No, that's unlikely. PHP's main advantage is its ease of deployment on shared web hosts: upload your files and you're done.

</comment><comment><USER31> This article has unresizable font as the main font. How am I even supposed to read that?</comment>


On Distributing Command Line Applications: Why I Switched From Ruby to Go - <AUTHOR> http://codegangsta.io/blog/2013/07/21/creating-cli-applications-in-go/

<comment><USER3> As a non-Rubyist, I share your pain of having to use Rubygems (and pypi, and CPAN, and all the others). However:

"Since Go is a compiled language, binaries of your app can be precompiled for each platform that you wish to distribute for."

For me, this is worse. There's no audit trail. No way for me to verify that the binary of your app is the same as the source you provide. So there's no way I'm running your binary on my system. I simply cannot trust it with my user account.

Not only is this bad from a security perspective, it's also bad for maintenance. Over time, many end up in a situation where others cannot rebuild binaries from source at all, since binary distribution becomes the norm and the source->binary mechanism isn't maintained except on the developer's own system. Look at the Java community for an example of this. This is a problem because it means that others cannot easy be a member of the community working on the source.

If you want to promote a community around the source (like Github does), you must also provide an easy way to rebuild binaries from the source.

Unfortunately, there's no good answer here.

Distributions like Debian have solved this problem - there are source packages, and there's a standard way of building them to get the binary packages. But there's quite a bit of arcane crud that's built up over the years, making it difficult to learn and understand Debian packaging. And the solution isn't cross platform.

Each language community has produced their own solution (Rubygems, Python eggs, CPAN, etc) but none of them work with the packaging provided by any other language. To me, this may be cross platform, but not being cross language is just as big a problem.

<comment><USER4> Nothing is stopping you from building the app from source instead, just like every other open source project. But at least with Go you have the _option_ of just downloading and running a binary if you trust the author.

<comment><USER3> This is true. I do have that option.

But I very rarely trust the author. Most of the time I've never heard of the author. This is a good thing, and exactly why Github has been so successful. By nature, if there's something obscure that I need doing, and I find some software or software module that lets me do it, then chances are I won't know the author.

This means that downloading and running the binary is suddenly a very obscure use case. It should be for you, too.

<comment><USER5> > This means that downloading and running the binary is suddenly a very > obscure use case. It should be for you, too.

A rare use case for you, but definitely not an obscure one in general. Besides, trusting the author, in this case, only matters if you're willing to audit the source to verify that nothing malicious is occurring -- something I'm not going to do in general. I mean, realistically, when was the last time you audited your network drivers? your OS? your browser?

<comment><USER3> You don't have to verify the source yourself. Others can do it. I trust the Debian maintainer for my package. He's a real world person who has had his key signed by other Debian developers.

The fact that the source can be verified also reduces the risk of malicious modification.

It's not necessarily just the author who might be malicious, either. The author's computer where he runs his compiler might be compromised.

</comment></comment></comment></comment></comment><comment><USER6> Not just command line apps. Any time you need to deploy on multiple machines, Go saves you a lot of trouble.

At work we use primarily `buildout` [1] to deploy Python apps with complex dependencies. It is rather slow and fragile. Additionally, PyPi and various external repositories are not very reliable, and we ended up setting up local mirrors. Nevertheless, the process is not very pleasant.

We are experimenting with Go in some smaller projects, and so far the experience is great! We can build on a development machine (OS X) with all the dependencies version controlled (using git-subtree) along with our code, and cross-compile a static binary for Linux for deployment. Deploying is simply copying the binary to the production machines (usual configuration management is done via Puppet).

Besides, Go can efficiently make use of multiple cores (remember to set GOMAXPROCS correctly), and we can just deploy one instance of an app per server, unlike one instance per core as in Python's case (I know there's multiprocessing, but it's bad for operation and management). This greatly reduces memory overhead if the app is big (e.g. consumes a couple hundred of MB upfront and you have 32 cores per machine = a few GB's memory is wasted).

Only problem is that Go's 3rd-party libs are not as comprehensive as Python's yet, and many packages we rely on do not have mature/reliable counterparts in Go. So we are still using Python for our core stuff, but smaller projects that do not require many external packages can be done in Go elegantly.

[1]: [http://www.buildout.org/en/latest/](http://www.buildout.org/en/latest/)

</comment><comment><USER7> He leaves out discussion of a crucial part of app distribution: dynamic libraries.

While you can't create and link with dynamic libraries in Go, you can call into C or C++ libraries. Go binaries are self-contained as long as all your code is Go. But if you're depending on a C or C++ library, you still have to worry about this aspect of distribution.

<comment><USER8> It is now possible to statically compile+link C libraries into your Go binary. The sqlite3 package ([http://godoc.org/code.google.com/p/go- sqlite/go1/sqlite3](http://godoc.org/code.google.com/p/go-sqlite/go1/sqlite3)) is a good example of this.

<comment><USER9> Another aspect is that some opensource licensing schemes may require you to link dynamically. The LGPL, for example, requires you to do this unless you release the rest of your source code.

</comment></comment></comment><comment><USER10> Blog author here. Long story short I got frustrated with distributing Ruby cli apps and decided to switch to Go for speed and distribution.

I also created a nice cli library built on top of the Go flag package. I've been having a lot of fun in Go so far and encourage anyone who hasn't given it a shot to try it out.

[https://github.com/codegangsta/cli](https://github.com/codegangsta/cli)

<comment><USER11> Made the same decision as you recently.

The only pep-peeve of mine is that I'm not a huge fan of using JSON as a configuration mechanism. I much preferred having a Ruby file for configuration. My middle ground here is using a system ruby via /usr/bin/env ruby and generating the necessary JSON output for people that want the nice syntax.

</comment></comment><comment><USER12> This also makes go applications trivial to deploy. What used to be a massive amount of plumbing and bootstrapping for a ruby environment with multiple app server instances behind a reverse proxy with all the associated configuration and dependencies can now be little more than pushing a binary to a (nearly) stock machine instance. It really simplifies ops.

<comment><USER13> I really don't see what having multiple app server instances behind reverse proxy has anything to do with Ruby vs Go. App servers behind reverse proxy is an architectural decision. You can write a Go web app that must be put behind a reverse proxy, or you can write a Go web app that is supposed to act as its own web server. Likewise, your Ruby app does not _have_ to run multiple app server instances behind a reverse proxy. You can run it together with your main web server (Apache or Nginx) and you can run a single instance instead. See [https://www.phusionpassenger.com/](https://www.phusionpassenger.com/)

Of course, installing dependencies still matters. For Ruby, you can just use the Brightbox Debian/Ubuntu packages.

<comment><USER12> All the mainstream ruby web frameworks require something like nginx in front of multiple instances of something like thin. You can use something like phusion, unicorn or rainbows that do forking for you instead of launching multiple processes by hand, but you still have to configure them and they're still running multiple app instances.

Go's built-in http library, and all the associated frameworks, run multi- threaded out of the box, and are fast enough that you don't need to put nginx or haproxy in front of them.

I love ruby, I've been building and deploying rails apps since 1.0, the ease of deploying go apps is precisely the type of thing the Ruby/Rails community cares a lot about: making developers' lives easier.

<comment><USER13> It is not true that they must run multiple instances. Rails has been multithreading-capable for years. Other Ruby frameworks have been multithreading-capable for much longer. With Phusion Passenger Enterprise, not only do you not have to setup reverse proxying, the sole instance can run multithreaded.

As for "fast enough that you don't need to put nginx or haproxy in front of them" \- the point of reverse proxying is not performance, it's security. If anything, reverse proxying makes things _slower_ (theoretically) because the kernel has to make one extra copy of the data. You reverse proxy stuff so that nginx or haproxy can properly handle I/O management and concurrency, and so they can sanitize HTTP headers, not because it gives you a performance boost. If Go frameworks are multithreaded by default, instead of events, then Go too can benefit from reverse proxying.

If you don't want to think about any kind of reverse proxy setup, there's Passenger Standalone in the Ruby world. You type 'passenger start', and you have a fully-functional, production ready server listening on any port you, that does not require reverse proxying.

<comment><USER12> "the point of reverse proxying is not performance, it's security."

Not true. You absolutely need forking/multiple processes on rails deployments to enable even a small number of concurrent requests. If phusion wasn't forking and load balancing between those running instances, your performance would be far lower.

EDIT: See this article from the Passenger folks on tuning a passenger deployment, it should be more clear after reading that how Passenger is dealing with concurrency (by running multiple app instances and load balancing between them): [http://blog.phusion.nl/2013/03/12/tuning-phusion- passengers-...](http://blog.phusion.nl/2013/03/12/tuning-phusion-passengers- concurrency-settings/)

<comment><USER13> I wrote that article. :)

The concurrency tuning explained in that article has got nothing to do with reverse proxying. You don't _need_ multiple processes either: the article says that you can use multithreading, and especially in the case of I/O-bound apps, that's enough. If you want to leverage multiple CPU cores however, that's a different story, but it depends on your Ruby implementation. If you use JRuby or Rubinius then threads are enough. But that is still unrelated to the question of whether to reverse proxy or not.

</comment></comment></comment></comment></comment></comment><comment><USER14> I agree that the self-contained nature of compiled Go programs is one of Go's most appealing characteristics. In fact, I think you were too diplomatic toward Ruby, and most other popular languages. In my opinion, to insist that distributing a program should not be easy is to expect too little of our tools. I can identify two main deficiencies that make it hard to distribute programs:

1\. Lack of a module system that scales well to large programs while still using static linking. According to the talk "Go at Google: Language design in the service of software engineering" ([http://talks.golang.org/2012/splash.article](http://talks.golang.org/2012/splash.article)), this is one of the main problems that the Go developers aimed to solve.

2\. Too much dynamism in the language, such that a packaging tool can't tell with certainty what a program needs, and just as important, what it doesn't need. Go is static enough that this simply doesn't apply. Dart is more static than JavaScript, but still allows some dynamism. The Google Closure tools solve the problem by subsetting JavaScript. Is there any such subset for Python or Ruby? (I don't count RPython, since AFAIK that's only intended for PyPy itself.)

So when choosing a language for a program that one expects to distribute to many users, I think it's a good idea to take these things into account. This may mean choosing a language that lacks some dynamic features that many of us find convenient. (Compile-time metaprogramming can compensate; too bad neither Go nor Dart has it.) Go looks like a pretty good language for servers and command-line utilities.

</comment><comment><USER15> I've written about this before, but this guy definitely wrote it better. I experience a very close relation between this and Ruby vs C#(Mono). I love Ruby as a language, but the pain of using Rubygems and random things requiring Ruby <1.9 and <1.8 (And setting up rvm to manage that) is an absolute pain. Python is slightly better, but I still have a python2 and python3 installed.

This is why I love mono. Things may not be backwards compatible, but I've never had a problem where something wasn't forwards compatible. And I think this is a result of it being compiled. The compilation step decouples the language and syntax from the actual execution of the code. This means that compiled applications can have drastic and breaking syntax improvements, but still be forward compatible on future runtimes. The same can not be said of interpreted languages unless you somehow work a version number into the execution, and even then it makes implementation of the languages much harder because future versions must support every version of the syntax

</comment><comment><USER16> Related to cli and Go, if you ever want to quickly execute Go funcs from a terminal [1]:

        $ goe go/parser 'ParseExpr("5 + 7")'

[1] [https://github.com/shurcooL/goe#examples](https://github.com/shurcooL/goe#examples)

<comment><USER17> What it needs is a `-e` flag:

        ruby -e 'puts 5 + 7'

<comment><USER16> Can you please elaborate on how that would help?

Is it just for familiarity/similarity? I kinda see the 'e' from ruby -e going after the go to form goe. goe -e would be just redundant, wouldn't it?

<comment><USER17> Perl, Ruby, Python, and many others have a succinct "evaluate this string" command-line flag. It's so commonplace it's to be expected.

</comment></comment></comment></comment><comment><USER18> _After attempting the distribution of a command line application via Rubygems I came to the following conclusion: Any application not intended exculsively for Rubyists should not be distributed via Rubygems._

Another option could also be to convert gems (and dependency gems) to deb or rpm packages [https://github.com/jordansissel/fpm/wiki/ConvertingGems](https://github.com/jordansissel/fpm/wiki/ConvertingGems)

As far as I know, the Foreman project utilizes this approach [http://deb.theforeman.org/pool/precise/stable/f/foreman/](http://deb.theforeman.org/pool/precise/stable/f/foreman/)

<comment><USER10> That is definitely an option, and it is essentially what we did for our linux support when we were building out a cross platform toolchain for mobile games. The tricky part was supporting Windows, Linux, and Mac all at once. Again, not suprised that distribution was hard, there were just a couple things that made distribution in Ruby a more painful experience

</comment></comment><comment><USER19> In a prior job, I did a decent amount of Python distributions. I had it simple: 1-2 python libraries plugging into a single app which was distributed on a .deb server. Even that was not particularly fun. I can _definitely_ understand the appeal of a build n run system.

As a user, I find rubygem based installs rather horrific (same for pypi & cpan).

n.b.: I'm sure there are tools out there that will appropriately massage your app and put together a app.zip package for you. But IME those sorts of tools are a lot of work to get going.

</comment><comment><USER20> One feature I miss from distributing on RubyGems or PyPI is easy update functionality. Thankfully, go-update[1] looks to solve the problem by making it easy to create self-updating Go binaries.

[1]: [https://github.com/inconshreveable/go- update](https://github.com/inconshreveable/go-update)

</comment><comment><USER21> How many architectures are officially supported by Go?

<comment><USER10> Supported architectures listed here: [http://golang.org/doc/install#requirements](http://golang.org/doc/install#requirements)

<comment><USER22> There are two implementations of go: gc and gccgo. The link above describes gc. gccgo is built on gcc and supports many more environments, at least in theory. gccgo will run on addtional processors like powerpc, mips, and sparc and other operating systems like solaris. These aren't necessarily well tested though.

</comment></comment></comment><comment><USER23> Why don't compile Ruby using f.e. LLVM?

<comment><USER24> Because it's kinda pointless. The thing that makes Ruby (very) slow isn't the runtime (there might be at most a 1.5x - 2x speedup there), it's the language design. All that dynamism and runtime reflection is really a performance killer.

<comment><USER25> _it 's the language design. All that dynamism and runtime reflection is really a performance killer._

So why are Lisp and Smalltalk, which are equally dynamic and reflective, even faster?

<comment><USER26> Ruby does have support for callcc, which most lisps, specially the fast ones like Common Lisp, don't, since it is a pain to implement a fast one without implications in your whole language. Can't comment on smalltalk, since I never programmed in it.

<comment><USER25> _Ruby does have support for callcc, which most lisps, specially the fast ones like Common Lisp, don 't, since it is a pain to implement a fast one without implications in your whole language._

Gambit-C is one of the fastest Lisp implementations around, and it _does_ have a full and performant implementations of call/cc. And I'm afraid that Common Lisp happens to be "fast by sccident" rather than "fast by design" \- there's probably more extra speed to be gained by having a better implementable standard (which CL isn't, or at least not that much) than by omitting call/cc.

(Also note that call/cc is sort of out of favour these days, delimited continuations is where it's at.)

<comment><USER27> Well, actually speed was a design goal for Common Lisp. Thus the standard defines the semantics of a compiler, including a file compiler. The CL standard provides stuff like stack allocation, inlining, fast function calling, compiler macros, type system, type declarations, compiler strategy annotations, lots of primitive constructs (like GO in TAGBODY). Most real implementations add several more speed-relevant features.

If you say that Common Lisp 'happens to be fast by accident', then this is completely misleading. 'fast' was a design goal and many implementors have done quite a lot to make demanding applications (like huge object-oriented CAD systems) to be fast.

<comment><USER25> _Well, actually speed was a design goal for Common Lisp._

It might have been one of the design goals, but it wasn't the ultimate design goal (which, I believe, was making a convergent compromise dialect to rule them all), otherwise certain parts of the language would have been most likely simpler. The things you mention are the good stuff. The problem is that there's also some bad stuff that hindered performance of CL on stock hardware until quality compilers were written. Yes, SBCL is quite fast. But you'd have a hard time trying to convince me that even modern implementations of Common Lisp represent some sort of performance optimum in the same way that, say, Stalin Scheme represents an optimum.

Also, wouldn't TAGBODY/GO be replaceable in presence of proper tail recursion by structuring the "TAGBODY basic blocks" into mutually recursive function calls? The same argument could be applied to inlining: Yes, the Common Lisp standard mentions inlining. But did you notice how nowadays, that's more or less considered an implementation issue rather than a language issue? How JVMs are capable of inlining just fine without having to be hinted what to inline and what not to inline?

Now I'd understand some of the hints, perhaps DYNAMIC-EXTENT makes somewhat more sense if the purpose is to say "It's my intent to create this value for a limited period of time, yell at me if I pass it out", but in general, from where I stand, a lot of the design decisions seem to make writing Common Lisp compilers actually more complicated to modern compiler writers.

<comment><USER27> Common Lisp had several design goals. Speed was an important design goal. The users of Common Lisp wanted to be able to deploy and run large and demanding software on stock hardware.

> ultimate design goal (which, I believe, was making a convergent compromise > dialect to rule them all), otherwise certain parts of the language would > have been most likely simpler.

No, there was no ultimate design goal and Common Lisp was not designed as a compromise dialect at all. Common Lisp was designed to be a modern replacement for Maclisp and was incorporating several projects working on a modern Maclisp variant, for example NIL, Spicelisp and Zetalisp. Common Lisp was not designed to compromise over Emacs Lisp, Franz Lisp, Portable Standard Lisp, Lelisp, UCI Lisp, Interlisp, Scheme and whatever dialect was there at that time (Common Lisp first round of design was between 1982 and 1984. Then the standardization of ANSI CL was worked on until the early 90s.) If Common Lisp was compromise, then mostly between Maclisp successors. Later during standardization the Condition System of ZetaLisp was integrated, the LOOP of Maclisp and CLOS was developed as a new object system (based on New Flavors from Zetalisp and Portable Common Loops from Xerox). CLOS was also not as a compromise, but a radically new system optionally based on a Meta-Object System.

CLtL1 says: Common Lisp is a new dialect of Lisp, a successor to Maclisp. Influenced strongly by ZetaLisp and to some extent by Scheme and Interlisp.

> that hindered performance of CL on stock hardware until quality compilers > were written

Quality compilers were written almost from day one: CMUCL, Allegro CL, LispWorks, Lucid CL. Of those Lucid CL was the fastest with the most optimizations.

Note that there are two or three different types of speed that needs to be addressed in CL: 1) the speed of unoptimized, fully dynamic and safe Lisp. 2) the speed of optimized, non-dynamic, and possibly unsafe Lisp. 3) the speed of production applications which need to be safe, somewhat dynamic, but optimized.

> Also, wouldn't TAGBODY/GO be replaceable in presence of proper tail > recursion by structuring the "TAGBODY basic blocks" into mutually recursive > function calls?

TCO is done by some implementations. Generally it is not seen as a useful default feature. Common Lisp favors iteration over TCO-based recursion. That's also what I favor. Since there are some implementation targets which don't support TCO (or where TCO makes implementations more difficult, one went to the simpler language). For example some of the Lisp Machines at that time did not support TCO. Today for example the JVM (where Common Lisp also runs on) does not support TCO. Scheme is seen as a different Lisp dialect and CL mostly learned from Scheme the default lexical binding - not more. Remember Scheme is much older than CL. CL did not adopt Scheme's one namespace, its calling semantics and conventions, not its naming conventions, not CALL/CC, not its macro system, ... This again shows that CL is not a compromise dialect - <USER2> simply does not support a lot Scheme-style programming - even though Scheme is a decade older than CL. By design.

> Yes, the Common Lisp standard mentions inlining. But did you notice how > nowadays, that's more or less considered an implementation issue rather than > a language issue? How JVMs are capable of inlining just fine without having > to be hinted what to inline and what not to inline?

Common Lisp was not designed for one special implementation platform. Some CL compilers may honor INLINE declarations (many do), others do inlining automatically (like Allegro CL). If an implementation of Common Lisp on the JVM uses its inlining capabilities, fine. But there are many other implementations. ECL for example compiles to C and nobody is would say it is a good thing to stop working on that, just because the JVM exists. The JVM is a language issue. It's the Java Virtual Machine. It was not designed to host Lisp or support efficient implementations of Lisp. One can implement Lisp on the JVM, but it is a not so good fit.

> a lot of the design decisions seem to make writing Common Lisp compilers > actually more complicated to modern compiler writers.

You are vague on these issues. Sure a good Common Lisp compiler is complex, but because the complexity is in things like the type system, type inferencing, etc etc. The base language itself is relatively simple. The result is that CL compilers still are producing faster code than most dynamic languages. Native implementations are still much faster and tighter than what the JVM can offer.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER28> What is the DSL used in the Ruby example?

</comment><comment><USER29> What's wrong with distributing source + makefile?

</comment><comment><USER30> I think todays programmers is doing everything back ways. First they think about what language they want to use, then what libraries do they need, and then finally on what platform will it run and how will it be deployed.

In the old days everything was much easier. First you chose a platform, depending on budget, support and so on. When you decided what platform to use, that then dictated what libraries you had at hands, and what package manager you needed for deployment.

That maybe sounds very limiting, but on the other hand you don't always have to come up with new solutions how to bundle and deploy stuff all the time...

<comment><USER13> Programmers "today"? Maybe I'm not old enough but I cannot remember a time when programmers would choose a platform based on their target userbase, unless they are forced to do so (e.g. iOS). Even back in 2000, given the choice, programmers would always choose to use their favorite language if they can. I remember that back in the days I was very much concerned about distribution. If I write an app in Java, users needed to download a 20 MB runtime (mind you, circa 2000 dialup was common). Visual Basic, Visual C++, etc all required some form of runtime. .NET was in its infancy and not widely distributed. I chose to write as much as possible in Delphi so that users only had 1 executable to worry about, even when Delphi wasn't the most productive platform. I asked questions on various developer forums, and I was being ridiculed for thinking so much about user distribution; the consensus from the community was: pick a language they like, and the user will just have to put up with whatever additional multi-megabyte runtime it requires.

<comment><USER30> Back in 2000? Good tools like Make where built in the 70s that's utilized by Autotools and other great build systems to bridge the gap between different platforms, so that you don't have to distribute as the author of the article we're discussing, a whole runtime with all dependencies.

<comment><USER31> But the main point of the article was (which I strongly agree with) that running stuff like Autotools, gems, or package manager kind of sucks when you just want to install a console tool.

At least back in the "old times" we were distributing program + additional runtime, and not requiring a random unexpected runtime in proper version + whole supporting infrastructure + program source code to build. The former option greatly simplified installation.</comment></comment></comment></comment>


Containers vs. Config Management - <AUTHOR> https://blog.containership.io/containers-vs-config-management-e64cbb744a94 <comment><USER3> This kinda makes the mistake that nearly everyone makes with regard to containers: They view the container as a black box into which complexity disappears.

A container doesn't consume complexity and emit order. The complexity is still in there; you still have to build your containers in a way that is replicable, reliable, and automatable. I'm not necessarily saying configuration management is the only way to address that complexity, but it _does_ have to be addressed in a container-based environment.

Now, I understand that in many cases some of the complexity is now being outsourced to others who maintain the containers (in the same way we've been outsourcing package management to operating system vendors for a couple decades), and so maybe some of the complexity has been pushed outside of your organization, but you have _something_ being deployed that is internal to your org, so you have to deal with that complexity. Container management tools just don't do things at that level.

There's always a point where what you're doing is unique to your environment, and you have to be able to manage/reproduce/upgrade it somehow.

<comment><USER4> > Now, I understand that in many cases some of the complexity is now being > outsourced to others who maintain the containers

We have a winner.

Containers in the end looks to management and developers as a means of outsourcing those pesky sysadmins that always get in the way of dreaming big.

Its not without reason why devops and containerization seems to go hand in hand, as both are pretty much about neutering or sidelining the sysadmin role.

<comment><USER5> The sysadmin role isn't being neutered or sidelined, it is being moved from something every company has to do to something that primarily will be performed by large providers at massive scale. System infrastructure and the management of it is becoming commoditized, and containers are one part of that story, just as the increasing level of abstraction of cloud platforms is another.

</comment></comment></comment><comment><USER6> This article reads as very developer-centric, is my first thought. When comparing Docker and Configuration Management, he uses deploying code and scripts as his comparison, which is far from extensive when thinking through a list of all changes on a system.

As a former operations engineer himself, the author didn't touch on a suite of other day to day complexities to consider : how to handle emergency change management and track that (most configuration management tools can revert to baseline if they detect a change they don't expect), keeping systems patched, etc.

While not a bad article, I feel like we've talked about just deployment, and quite frankly there are a zillion ways to solve the automation of it.

<comment><AUTHOR> That's fair. This was definitely written with deployments in mind so I can't disagree with your assessment. With that said, I think the real downside of config management, and the real upside of containers really comes down to deployments. There is definitely still a use case for system level changes and patching that are best served by config management. How long until the container based config management? :serious_question:

<comment><USER7> It's already possible to use config management tools to manipulate container images. E.g. Ansible 2.0 has a Docker connector.

I talked about it a bit here: [http://go- talks.appspot.com/github.com/gigablah/talks/201509...](http://go- talks.appspot.com/github.com/gigablah/talks/20150915/from_source_code_to_containers.slide#23)

Judging from the horrified reactions of my audience though, I guess it's not a popular idea at the moment :p

<comment><USER8> Why were people horrified? I liked the points you made in your slides, specifically patching openssl in containers is a great use case. When will this connector be beta? By the way this connector along with something like "clair" would be a good foundation for vulnerability management system for containers in production:

[https://github.com/coreos/clair](https://github.com/coreos/clair)

<comment><USER7> The connector is already out, Ansible 2.0 was released back in January.

</comment></comment></comment></comment></comment><comment><USER9> I see some parallels between J2EE and containers. In both cases it is a platform that runs a "bundled application" with the hope of interoperability and ease of use. Download and deploy the .war (and edit five .xml config files) and you're done. Pull the image and run CMD in it, and you're done.

No matter how good the "easy app runner" platform is, one still has to specify the network between apps, logical dependencies, and --volume dependencies. And also "runtime config" to the individual apps. I really like the docker compose approach, but does a docker compose recipe work on every docker hosting provider? Also, the 12factor app approach is good, but is just a bunch of ENV vars sufficient for every config one might want to do for an app?

</comment><comment><USER10> When facing this duality a few years ago, I had to look at the facts that encourage Config Management in favour of a Dockerfile approach:

* Dockerfiles do not work for configuring host systems, only Cfg Management is applicable

* Configuration management systems usually have a very declarative approach, easier to extend and maintain that Dockerfiles and bash scripts

* Dockerfiles contain too many arbitrary choices that do not work for everyone, starting with the choice of the distribution and OS version: companies like standardizing around one distribution and would have to fork Dockerfiles based on other systems, every single time.

The best solution I see at the moment is to use setup containers using configuration management.

<comment><USER11> I'd add the Dockerfiles only allow extension by inheritance, leading to a lot of cases where people extend for construction.

Put another way: you can't safely compose Dockerfiles. If you want clean, narrow, special-purpose containers, you have two alternatives:

1\. Build Dockerfiles for each use case precisely. This will lead to duplicated code and all the excitement thereof.

2\. Build a giant shaky tower of Dockerfiles with FROMs, of which most are essentially 'abstract' Dockerfiles, not intended to be actually used. This will lead to documentation and all the excitement thereof.

Stackable filesystems are a triumph in some regards, but they create ugly backpressure on the design of libraries of Dockerfiles.

In this respect I see configuration management tools as vastly superior, as they generally have a concept of assembling goals and converging to them, rather than a concrete stacking logic.

<comment><USER12> 2 does not work for many scenarios. You end up having to build all the FROM combinations which quickly grows out of practicality.

3\. use any templating system you like

For example, I make mixins out of little bash scripts (mostly heredocs) that outputs Dockerfile lines. Mixins are shared between projects via git submodules. Voilà.

I don't like configuration management mainly because the syntax is plain stupid, making it difficult to read. Check out Ansible and you'll know what I mean... although I do I have some hope in Terraform once they roll out their feature of "write out the config files out of whatever I already have in that AWS account" (not having to write the config scripts from scratch gives me a tailored example to learn from).

Whenever some bash script gets big (say, 100LOC), it gets rewritten in ruby.

<comment><USER11> I feel that writing a code-generator for Dockerfiles is a strong smell that they are insufficiently expressive for their task.

And nor should they be, given the unavoidable constraints of the design.

> _Whenever some bash script gets big (say, 100LOC), it gets rewritten in > ruby._

100? You have patience I lack.

<comment><USER12> I think of Dockerfiles as a very static piece of code, it urges you to use templates on it.

Just like for a nginx.conf

And 100LOC, well, I've gotten very much used to bash. It is so comfortable when the terminal is your REPL. To tell you the truth my main deploy script is actually 500LOC of bash. It reads top to bottom with a lot of conventions (I was suprised to follow google's bash conventions), and short circuiting abuse.

</comment></comment></comment></comment></comment><comment><USER13> The author seems to think configuration management is about installing packages. That's something the package subsystem does, with a reasonably mature handling of dependencies too. That makes it look like a glorified shell script. (Which may be a case of when all you've got is a hammer everything looks like a nail.)

But configuration management concerns centralized management (hence "management") of decentralized systems. It can answer questions such as "why is this sceduled job running on nodes of type x but not y and who put it there?", and guarantees such as "application w and z are always in lock step concerning parameter p".

I've seen organizations move to containers, and they all inevitably end up with more and more containers and increasing complexity. Centralized configuration management is more important in that environment, not less. Modern tools such as Ansible and Puppet have grown up in a devops worls and have good support for managing containers (even if they are a bit of a moving target) and there is no reason to be scared of them.

</comment><comment><USER14> > All of the logic that used to live in your cookbooks/playbooks/manifests/etc > now lives in a Dockerfile that resides directly in the repository for the > application it is designed to build. This means that things are much more > organized and the person managing the git repo can also modify and test the > automation for the app.

It sounds like the author didn't have experience with larger systems - <USER2> maybe did, but my experience contradicts this.

Let's say you have everything you can in the containers. Now you want to deploy test and production environments. How do containers know which environment they're running in? Or specifically, things like what's the database user/password, what queue to connect to, where to find local endpoint for service X?

That still needs to live outside of containers. And at some point etcd and similar solutions have a problem of "what if I don't want to share all the data with all the infrastructure all the time"? Well... you can solve that with a config management service. Edit: just noticed etcd actually gained ACL last year - good for them. But how do you configure/deploy etcd in the first place?

<comment><USER15> _It sounds like the author didn 't have experience with larger systems_

In actual _large_ systems like Facebook or Google you explicitly keep runtime applicaiton configuration info out of the config management system and use another system entirely to manage that. This lets you easily manage test/deploy, shard services easily, and do rolling deployments with config info updated instantly across the fleet if necessary.

</comment></comment><comment><USER16> > All of the dependencies of the application are bundled with the container > which means no need to build on the fly on every server during deployment. > This results in much faster deployments and rollbacks. ... Not having to > pull from git on every deployment eliminates the risk of a github outage > keeping you from being able to deploy.

Obviously there are other benefits, but it's funny how much of the motivation for containers comes from "now we don't have to do git clone && npm install!", which was always the case.

<comment><USER14> Eliminating running applications which talk over the network from deployment is a huge gain. Making things happen once rather than per-server is also a huge gain.

On one server, cloning and installing things from the internet is all fine. Installing/upgrading hundreds at a time means you get to see how unreliable package management can be. You also have to guarantee enough bandwidth to the repo/package server to handle everything, or implement orchestration so servers don't install the same thing at the same time. At scale this matters.

</comment></comment><comment><USER17> > _Instead you can write a block of code with Chef, for example, that > abstracts away the differences between distributions. You can execute this > same Chef recipe and it will work anywhere that a libxml2 package exists._

But this doesn't really work. What if the package is differently named on different distributions? What if one distribution's version of the package isn't compatible with your use of it? Besides, how often do you switch between distributions on your servers?

<comment><USER14> > What if the package is differently named on different distributions?

You write a case/when block that chooses the right name.

> What if one distribution's version of the package isn't compatible with your > use of it?

Rarely happens, but then you just compile/package your own. Or solve the incompatibility from the other side.

> Besides, how often do you switch between distributions on your servers?

Rarely. More often you want to reuse configuration management code from a common repository which installs X, but abstracts the distro differences because many people collaborated on it. Also, often you're running multiple distros within the same company but it helps to have the same deploy/management process.

(all answers from personal experience)

<comment><USER17> But then it's not really abstract. Not that it really matters, I just thought it was a strange thing to highlight.

<comment><USER14> Abstract / concrete is a spectrum ;) It abstracts over all the things which are standardised, known ahead of time, and known at the time of writing the system. Things that people just come up with are always going to be unstable.

It's not that big of an issue in practice though. About the only package I can remember out of under a hundred I dealt with was mysqld/mysql-server. Names match up in almost every case, especially on user-facing things (less on -dev/-devel and similar ones)

</comment></comment></comment></comment><comment><USER18> I decided to use docker since a particular client I am working with mandated that we use bare metal servers they own. A problem that I haven't quite solved yet is that I have several distinct hosts running docker daemons, with certain apps designated for certain hosts. I wish there were a way for docker compose to know about multiple docker hosts. Docker swarm could maybe help, but the clients specifically wants certain things to be redundant across two specific hosts.

<comment><USER19> Would kubernetes be too much overhead? That would be able to meet all of your needs.

</comment></comment><comment><USER20> No mention of how to debug code in containers... or shared containers created so developers can share libraries.... or a multitude of other things which do happen when you start letting developers directly push things to production

If you are using docker you need to ensure the docker container author takes the pager for the services he provides :)

</comment><comment><USER21> I a container layer even needed if you use a system like NixOS/NixOps for configuration management?

<comment><USER22> Is NixOs ready for prime time? I came away a bit dismayed after playing with it a while last weekend.

[https://github.com/NixOS/nixpkgs/issues/16321](https://github.com/NixOS/nixpkgs/issues/16321)

</comment></comment><comment><USER23> > Not having to pull from git on every deployment eliminates the risk of a > github outage keeping you from being able to deploy. Of course it's still > possible to run into this issue if you rely on DockerHub or another hosted > image registry.

lol

</comment><comment><USER24> Link is obfuscated and shortened. Should be changed to target URL: [https://blog.containership.io/containers-vs-config- managemen...](https://blog.containership.io/containers-vs-config- management-e64cbb744a94)

<comment><USER25> Thanks, we updated the link.</comment></comment>