Discover Flask - <AUTHOR> http://discoverflask.com/ <comment><USER2> It's interesting (exciting) to see all of the excitement about Python frameworks on HN lately. I've used both Flask and Django extensively, and highly recommend them.

I used to be on the Flask "micro-framework" bandwagon...until I realized that almost everything I was integrating into Flask was already included in Django (e.g. a test suite, an ORM, database migrations, authentication, an admin interface, etc.). Don't get me wrong; I love Flask. It launched me into web development with Python at a time when Django only confused me. But I just can't justify using Flask for a big project anymore when Django is just so much faster to work with (for me). If I need to, I can customize nearly any part of Django I want, store it as a project template, and use `django-admin startproject myproject --template="[https://sometemplate.com/in/a/git/repo.git"](https://sometemplate.com/in/a/git/repo.git") ` to start a new project with it.

<comment><USER3> Django is cool, and has been my choice for a fair number of projects. Although recently I had an interesting experience with it when I wanted to use websockets (yes, I know it's not written for that use case).

It was stunning that most of the solutions out there rely on some sort of a hack to get it working. I ended up porting over my project to Tornado, because I need websockets for a lot of the features I'm working on. And it worked out really well, especially after I combined it with SQLAlchemy and Alembic. 80% of the feature set right there. And SQLAlchemy is such a fantastic toolkit.

I'm hoping that django channels ([http://channels.readthedocs.org/en/latest/](http://channels.readthedocs.org/en/latest/)) stabilizes soon.

<comment><USER4> I prefer to split a server app into a connection management layer and a backend logic layer. This way the backend can remain stateless which makes for easier maintenance. I use Django as the backend piece in most of my realtime apps (for example WebhookInbox) and it works great.

Of course, native support for long-lived connections in Django would be cool, if only because it would mean the connection management layer could be built with Django too.

</comment></comment></comment><comment><USER5> Great job. I recommend Flask to everyone who wants a clean and modern framework that is simple to use and manage/deploy.

I'm also planning to write a series on how to build a simple SAAS with flask, hope some will find that interesting.

<comment><USER6> Flask is using globals everywhere, and for very long time did not support Python 3.x.

<comment><USER7> Although most of the methods are globally bound to the current request, Flask is thread safe and I can't see any problem with it. (except that it encourages beginners to use globals, which might be process and thread-unsafe).

<comment><USER8> I really don't see any benefit or sense in the Flask way of exposing request state through imported globals. It makes testing of views unnecessarily complex: you have to either patch the import or set up the thread local context in every test. And like you mentioned, it encourages unnecessary coupling.

It would be so much simpler to just have a request argument passed to each view.

<comment><USER9> But then you have to explicitly pass the context to everything you call.

Dynamic scoping would solve this problem (and similar problems, like dependency injection), but Python doesn't have that.

Flasks way emulates it as well as possible.

</comment></comment></comment></comment></comment><comment><USER10> Great work, but in my opinion using Flask for full-stack, end-to-end web development is wrong. A microframework should never be used as an end-all-be- all. By the time you're done you've added god knows how many external dependencies and bits of homegrown code to support "standard" features such as authentication, forms, etc. that are part of a proper full-stack (read: not micro) web framework. In this case, something like Django would be _much_ more suitable because by the time you've finished this Flask tutorial you've basically implemented key parts of the Django standard library.

</comment><comment><USER11> I found it very strange that the domain (discoverflask.com) is just a redirection to the Github repository.

Anyway, great content for Flask beginner. It's a wonderful framework for anybody wanting to write clean APIs and it's this kind of content that makes the community awesome.

<comment><USER12> I've done that redirect-to-github thing in the past. Usually, the intention is that the redirect is intended to be a temporary placeholder until I can create a "real" site at the domain.

</comment></comment><comment><USER13> I've been very happy with Bottle. [http://bottlepy.org](http://bottlepy.org) Pretty similar, but lighter.

<comment><USER14> Glad to see Bottle is still alive. I contributed to it a tiny bit a few years back, and it was definitely a fun too. I ended up moving to Flask because the larger community and wider set of plugins and tutorials.

</comment></comment><comment><AUTHOR> And here's a link to the Vagrant instance

[https://github.com/apolloclark/discover-flask- vagrant](https://github.com/apolloclark/discover-flask-vagrant)

<comment><USER15> Great work. I'll definitely look into this one

</comment></comment><comment><USER16> This is great. For anyone interested in learning further, this is basically considered the Flask bible:

[http://blog.miguelgrinberg.com/post/the-flask-mega- tutorial-...](http://blog.miguelgrinberg.com/post/the-flask-mega-tutorial- part-i-hello-world)

</comment><comment><USER17> If the owner of the repo drops by, I'd just like to say thank you for pulling all this together. Side note, I'm definitely more of a fan of the blog post tutorials like with the first 2. It's nice to be able to easily skim through for what you need, the documentation is also nice and condensed like cliff notes of the actual documentation. I also like how it's accompanied by the videos too though, so if you're looking for a little more detail you can go check out the video really quick to see what other side comments or pointers are made in the vid. Yes one can just look at the documentation, especially with how great the Flask documentation is, but sometimes it's just easier to check it these tutorial styled helpers since they have certain objectives and extra commentary.

</comment><comment><USER18> I've played with a bunch of Python web frameworks and amongst the micro- frameworks, Flask is the one I've settled on. Larger community, more integration with other packages.

I've also recently been using Django, and I'd say the biggest advantage of Flask is simplicity.

When you get to something the scale of Django, there's a whole load of decisions and culture baked into the approach, and you have to learn that in order to use it.

For small projects, I'd recommend Flask.

</comment><comment><USER19> After reading some comments pinning Flask and Django for "big" projects, I feel compelled to write my own experience.

Flask is a micro-framework, but it's not a toy framework (anymore). I've used it professionally for _all_ web related projects big and small for the last 4 years with great success. I wholeheartedly recommend it. It's not for everybody though, as one can attest by various comments lamenting its minimalist approach. Things that attract someone like me who likes a certain degree of control on how my tools are set up and work, may be the same that repulse someone else more concerned with having everything available in the kitchen sink.

When I moved to Python for web development Django was the most common recommendation. I had toyed with it a bit previously and considered settling for it. Amid the raves though were a few complains that gave me pause, most coming from experienced Python programmers. A common remark was that as time went and as they evolved with the framework, it wasn't uncommon to swap parts out of it for external libraries. SQLAlchemy, WTForms, Jinja2. In some case basically ending with only the routing module and the admin. Which raised the question of the relevance of using a full-stack framework not necessarily designed with interchangeability in mind, if you end up just using it like a glue mini-framework. Having worked with full-stack frameworks before I identified with the highlighted pain points and Flask became very attractive. I picked it up for a first project and never looked back. It's been over 4 years now. It's entirely possible that I'm missing out by not working with something more like Django, although I doubt it very much.

Flask itself is small, made of bits of the Werkzeug WSGI toolkit and the Jinja2 template engine. It's simple in concept, simple to set up, and simple to use, but anyone who's been developing with it for anything more than basic projects (static sites, microblogs, etc) knows that to do more complicated stuff than displaying template pages you will need to add some dependencies (as it should be). You can do it the old-fashion Python way, by simply importing modules and working with them, or you can install an extension. There's currently a great variety of extensions, the official list is here [http://flask.pocoo.org/extensions/](http://flask.pocoo.org/extensions/), with lots more available from Github and PyPI. Furthermore it's dead simple to make your own if you're not satisfied with the offering. All of this is probably the reason why 5+ years after its creation you see a good number of guidelines on how to work with it for ambitious web projects.

</comment><comment><USER20> I haven't checked the quality, but I love the idea and presentation. This is an awesome way to approach showing people how to build fullstack websites in any framework or language to advanced beginner or intermediate programmers.

As someone who procrastinates on their own blog to teach people this topic, this is going to inspire my direction a lot.

</comment><comment><USER21> I don't know anything of flask or python really. Is Flask in python 2 or 3? Do I go learn python 2.7 right now or Python 3? Is stuff broken the the eco system. Does it make sense moving over from JavaScript for easier backend CRUD'ish app's? Thanks HN.

<comment><USER22> Python 2.7 will be around for a while due to legacy code, but it's only a matter of time before Python 3.x is more widely used. Don't invest time learning skills that will only dwindle in value.

Some stuff is broken in the ecosystem. That's true of all ecosystems. If you're coming from the JS world it's far, far better. But if you are importing every 0.x versioned library that vaguely solves a problem close to your problem, no ecosystem will save you.

Python is a solid choice of language for the back ends of web applications.

<comment><USER23> > Python is a solid choice of language for the back ends of web applications.

With caution advised when it comes to scaling. You might, or might not, pay a pretty hefty premium in the amount of hardware needed.

<comment><USER24> I don't want to come across as a Python zealot here, but in my experience, Python has almost never been the problem in scaling. Certainly not to the extent where I would say that it costs a premium in hardware. Though, I suppose it depends on the application.

My experience has been that Python will give you more than enough rope to hang yourself with, and lots of apps that are meant to scale have some homegrown crap of an architecture, or people try to get fancy when they don't need to and try to write go or scala in Python.

If you have a clear understanding of your data model, Python backends scale quite well vertically. Run one instance of your app per core you have available and use IPTables to round robin (or whatever. you can get fancier if you feel like it) to each app instance. Boom. You are now banging on all 4 or 8 or however many cores your server has without the context switch overhead of the multiprocessing module and also without the headaches that go along with the shared memory space of the multithreading libs. Not to mention cleaner, nicer code.

If you build your product to work well in this way, then scaling horizontally also becomes a non-issue as far as Python code is concerned. You just put haproxy in front of multiple machines, set up health checks, and for the vast majority of use cases, you're done until you need to scale the storage horizontally. Then you have some serious decisions to make. But that's a different topic.

But back to my original point: every case where I've seen Python being all slow has really been an artifact of a crap architecture, or bolting on crap front end frameworks, or depending too much on an ORM like SQLAlchemy without really understanding your data and depending on it to do everything for you instead of thinking rationally about what your data _is_. Or some combination of all three above.

Yes, I know that there are languages that execute much faster than Python. I work with C#/.NET/SQL Server almost as much as I use the Python/Pyramid/PostgreSQL stack. I can write a crappy, slow, messy architecture on either stack. And in fact, when I was first getting to know C# and still writing Python code in that language, I did that.

The bottom line is that if you understand the strengths, weaknesses, and idioms of the language you are working with right now, almost every high-level language in common use today is sufficient for massive-scale deployments for the most common types of web apps. Heavy computing or scientific purposes obviously excepted. But most web apps that scale to hundreds of millions of users are basically "window-on-data" type applications.

I can't really think of a case where some other language besides Python would be noticeably cheaper hardware-wise.

Maybe you have to know your entire stack a bit better if you want scalable Python apps, maybe you have to do a little more Jiggery-Pokery with your Linux servers, but if you are serving at massive scale, you are going to have to go there anyway, especially when you start dealing with storage. That's part of the game.

<comment><USER25> > Run one instance of your app per core you have available and use IPTables to > round robin (or whatever. you can get fancier if you feel like it) to each > app instance.

But if you're using Flask and you have 4 cores with 4 instances, and you're making 4 database requests at the same time you're whole application is suddenly locked. And it's not uncommon for a single view to make up to 4 separate database requests. Most modern web applications are I/O bound, therefore a single Node.js process would have better performance here.

<comment><USER26> Wait, in PROD, you don't use Python as the server, you stick uWSGI in front of python/flask, that will deal with the network I/O limitation.

<comment><USER25> How does uWSGI help here? The OP recommended running 4 instances of Flask/Python, and with uWSGI you still have 4 instances. It doesn't help with I/O.

<comment><USER26> Because with uWSGI you can mix a number of threads and processes (recommended because more processes than cores wastes time with context switches, and a low number of threads per process is a waste of cycle. uWSGi is an event loop (not unlike node) which makes dispatching requests to threads and processes very easy.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER27> Great documentation set-out.</comment>


Why do so many people continue to pursue doctorates? - <AUTHOR> http://www.theatlantic.com/education/archive/2016/04/bad-job-market-phds/479205/?single_page=true <comment><USER4> Having done so myself and observed lots of people do it:

1\. It's a way to (pointlessly) delay adulthood.

2\. Fear of the job market.

3\. Don't know what else to do.

4\. Magical thinking (despite the numerous articles out there, like mine ([http://jakeseliger.com/2012/05/22/what-you-should-know- befor...](http://jakeseliger.com/2012/05/22/what-you-should-know-before-you- start-grad-school-in-english-literature-the-economic-financial-and- opportunity-costs/)) that attempt to dissuade it). I think this is the biggest issue. In addition, there seems to be a Lake Woebegone effect: Everyone thinks _they 're_ going to be above average.

EDIT: I forgot to add another important one: Grad school is pretty easy and fun! You get to hang out on campus, think about ideas, take a minimal number of classes, do a bit of teaching, and have copious free time. The problem is that, as time advances and your priorities start changing (want a real life / job, date people who don't date people whose lives aren't together, etc.), reality starts to intrude. Many grad students have an unacknowledged Peter Pan complex.

<comment><USER5> I did really well when I finished my undergraduate program, and my professors kept trying to get me to pursue a Doctorate or PhD. in my field. They were really enthusiastic about it telling me that with my talents I deserve more than a B.A.

None of them could give me a convincing answer as to how I would benefit long- term for continuing my education. None of them could give me an example of a job that I would be unable to get with my current level of education. All of them admitted that it would require 4-6 more years of my life, and that even with the best financial aid / scholarships, I would still be paying $60-100k out-of-pocket for the experience.

So basically, it would have required me to put off getting a full time job to instead drive myself deep into debt all while sacrificing all of my free-time until I was 30.

Instead, I moved to Seattle, found a job in I.T. and I have been climbing the corporate ladder ever since, making more than enough money to live comfortably and save money. I was lucky enough to earn my B.A. without any student debt.

My degree isn't even in an I.T. field. I have a B.A. in Journalism! The thing that has done more to further my resume has been my full-time employment and my technical certifications. I get to expense my certifications to my company...

Furthering my education would have ruined my future. Ending my education has accelerated it. Sadly, I worry that this is going to become true of Undergrad programs in the next decade, and that's not a good thing...

<comment><USER6> In STEM, if you don't get a financial offer along with your PhD program acceptance letter, then you've probably just been politely rejected. The financial offer will usually cover your living costs (a few $k per month), so it won't cover any lost opportunity costs... but you shouldn't need to take on debt to do your PhD in STEM. I'm guessing journalism is a different story!

For STEM, it's also true that certain job opportunities (eg. elite R&D facilities) more-or-less require a PhD or equivalent levels of mastery. It's possible to obtain those jobs after a long period of field work (without PhD), but it's a lot more difficult. Plus, you will always be butting up against credentialing norms during those jobs' recruiting processes. (Whether that's equitable or not is another story.)

Just my $0.02.

<comment><USER7> > In STEM, if you don't get a financial offer along with your acceptance > letter, then you've actually been politely rejected.

This is important. I thought at the time that I accepted that it was due to the massive budget cuts happening in California, and the UC's were undergoing furloughs.

Oops.

<comment><USER8> A prof I know at Berkeley won't let anyone in the Ph.D. program unless he can secure funding for them. He's in a humanities major, though. Maybe in STEM profs think they can land a job to pay it off.

</comment></comment></comment></comment></comment><comment><USER9> I am actually going to get my PhD in the next few days.

Perhaps I am an outlier, but I really think it was the right thing for me. I hit the end of my undergraduate degree and I felt like I had just started learning, so I looked into graduate school and liked what I saw. I've spent the last 4 years working on my PhD and I have learned a lot, I've made real improvements to human knowledge, and I've gotten a job that would have been impossible with a bachelors and 4 years of work experience.

But the biggest reason is simple and terrifying: because it is fun.

<comment><USER10> >I've spent the last 4 years working on my PhD and I have learned a lot, I've made real improvements to human knowledge, and I've gotten a job that would have been impossible with a bachelors and 4 years of work experience.

The real problem with oversupply of PhDs is there are fields of study which aren't remunerative outside academia. If you get a PhD in electrical engineering, say, or something medical, you can get a job.

But if you field of study was sociology or literature and you can't find a job in academia (and you probably can't), you're never going to be able to pay of eight or ten year's worth of student loans.

<comment><USER11> A similar argument could be made for not studying any humanities subject. The point is often not what the degree is in but the soft skills you gain. A good undergrad degree is seen by many employers to be enough to get a job (outside of specific technical fields).

If you have a PhD it is important to sell the skills it gave you that an employer wants. A good list of which are found here: [https://careercenter.umich.edu/article/phd-transferable- skil...](https://careercenter.umich.edu/article/phd-transferable-skills)

</comment></comment></comment><comment><USER12> 4 years of focusing on one subject, going to conferences all over the world and coupling them to vacations there, working with smart people of the same age, working in a "students atmosphere" while getting paid, having the time to really find out how things work, picking courses you like... Why are not more people doing PhDs? (Or this is a Dutch perspective?) (Granted: I didn't have that publish or perish attitude that landed some of my colleagues in places like MIT, still I have a very nice job at the moment.)

Don't know how it is now but we used to work 38 hrs, the additional 2 were given as free time on top of the standard 25 days/year giving you about 7 weeks off each year. You have no kids yet (in general) so you can travel a bit (as said, often after a conference so the boss pays the ticket.) I had a blast.

I did have some problems finding a job afterwards but this was end of 2009 when companies here basically stopped hiring for some time. But I spend 8 months applying while getting 74% of my last paid wages as a PhD student. Still, that was not as fun as it sounds.

<comment><USER13> Yeah, that is definitely not the US perspective on it. Here you have to pay to go to school, unless you are one of the lucky few who gets some sort of research grant, or teaching position. In which case you get to live on basically minimum wage, and travel is a extremely rare occurrence. You have now spent 10+ years getting yourself into debt, rather than making any money beyond what it takes to survive, and your job prospects aren't greatly increased. Its not that great a deal here.

<comment><USER12> Wow 10+ years. Here the University receives a sum of money when you get your PhD, that motivates them quite a lot to get you to finish. Most people finish within 4-5 years in my experience. Getting a PhD is like a job here, I didn't know it actually cost money in the US, in that case I can certainly understand people doubting the added value. What I got out was about the value of taking control, my prof gave me quite some freedom. But here also sometimes PhD students are used a measurement slaves by some micro-managing UD (assistant Professor). And also here sometimes people quit but it is more because they are lost without progress after 2 years or so.

In the US: Market forces seem to diminish the value of a PhD. In the Netherlands: Market is ignored by subsidies from the government. No idea what is better, let it die or keep the PhD system alive artificially. I don't know, the whole system feels a bit archaic to me. I always tell people that IQ and level of education are hardly related. I think it is difficult to fail a PhD here at least it doesn't happen often, especially when you have a professor that says exactly what you should do, and then you do it. I never met a dumb PhD student though.

<comment><USER13> I should clarify, I meant its 10+ years total in school 4 for Bachelors, 2-3 for Masters, and 4+ for the PhD.

</comment></comment></comment></comment><comment><USER14> A Ph.D. is a union card for doing research. If you want to be a university professor, a principal investigator at a lab, a Chief Scientist or Chief Scientific Officer at a major corporation, then you need that union card.

A Ph.D. program should not be for people who just like learning more stuff. You can take classes, get a masters degree, or just study on your own, if that's what you want. A Ph.D. is supposed to be a certification that you can create an original work of new research, at a quality level that meets the approval of other professional researchers. The research project is typically substantial enough to require years of effort, and the support of others who have already gone through the process.

<comment><USER15> Thats the theory. Plenty of PhD students I see just do what their supervisor tells them and don't seem to be especially smart these days.

<comment><USER16> 7402 is saying it's necessary.

you are saying it's not sufficient.

you might both be right; they are different properties.

</comment></comment></comment><comment><USER11> I appreciate the article is written from a US perspective but here is how it works in STEM in the UK:

1\. Get a funded PhD offer for 3-4 years (The funding is tax free so comparable to the average grad salary).

2\. Complete in 3-4 years (it is hard to get more than a 6 month extension, technically they can take their money back if you do but this rarely happens as the system wants you to quit early or succeed).

3\. Be more employable (I'm speaking from personal experience here. I didn't go looking for jobs like I did after my undergrad, they came to me via LinkedIn/Monster etc.)

As an avid reader of PhD comics I have always struggled to understand why the US system is so different.

As a side note it is worth pointing out that it is foolish to assume that doing a PhD inherently leads to a Professorship now. What it does do is give you 3-4 years experience (which employers like) and a vast range of research and communication skills (which employers love).

<comment><USER17> I do agree with this comment. I work as a c++ programmer and my friends who stayed at uni to do their PhDs are bringing more money home each month than I do, mostly because it's tax free and they get paid a lot of money to do marking and demonstrating on top of their PhD stipend. And yes, you have to finish in 4 years, extensions are uncommon.

<comment><USER18> On the other hand though, if they cannot get on track for a role in academia, once they get a job in industry they'll likely make less than you because of lack of practical experience. In such an ageist marketplace as programming in particular, being older and inexperienced is really unattractive.

</comment></comment></comment><comment><USER19> What the article doesn't mention at all is that many people pursue a PhD not for financial reasons but because it (often) provides an opportunity to work on very interesting problems with very smart people, something which is just not possible in most industry jobs. And I don't mean this in a derogatory way, it's just that (non-applied) Academic research is something that is fundamentally different from company research/work because it is (usually) not money-driven and has a very long-term focus.

Not everything in life is about the money, otherwise all people would try to become investment bankers.

<comment><USER20> While some people certainly are motivated by interest in doing pure research, I don't think that's the dominant factor.

From my anecdotal experience, far more are turning to PhDs simply as a way to avoid the brutal job market. They're choosing it for financial reasons, even though their reasons are backed by faulty information.

</comment></comment><comment><USER21> I am not aware of a way to get into a "scientist" level position doing biology research, in industry, without a PhD. This alone would answer the question "Why do so many people continue to pursue doctorates?"

If somebody has evidence to the contrary, please fill me in.

<comment><USER22> I've spent the last couple of weeks intensively researching paths into the life sciences without a PhD (I'm deciding whether to go get one).

I haven't found a way in either. If you start your own company somewhere in the field, or otherwise support yourself, you can rock and roll as an indie researcher. Otherwise, a Ph.D. seems necessary.

I wonder if this will change over the next decade, as synthetic biology continues to expand rapidly.

<comment><USER23> There is no non-PhD pathway into biomed research that leads to career advancement beyond the drone level [1]. If you do get a PhD, there are few jobs, relative to the number of graduates. All in all, I'd advise against it, unless you have a very specific love of the field, and can handle a modest, academic lifestyle in exchange for working very hard at what you do. And for that, the drone-level jobs are perfect for testing the waters: you should get a job in industry first, and if you decide that you love it, get a PhD.

Also, beware the "$TOPIC is expanding rapidly" trap: synthetic biology is merely the buzzword of the moment. When I started, it was computational biology. Later, genomics. The number of opportunities created by these booms has never kept up with the hype waves that preceded them.

[1] There is, however, a pathway for "business" people who enter biomed. But I'm assuming you're not interested in this, and in any case, getting a PhD won't help you with it.

<comment><USER22> Thanks for the advice, that all sounds right to me. I particularly like the point about the $TOPIC of the moment.

The roll-your-own-company lab approach sounds a little better, overall. One tricky thing is that I'd have a lot more credibility starting a biotech company with a Ph.D. Right now I'm considering a Ph.D. to get the training and credibility (union card, as someone else said). Still not a slam dunk, though - Transcriptic's founder just has a bachelor's.

</comment></comment></comment></comment><comment><USER24> I did a PhD because I really loved my subject (mathematics). I never envisaged it being a path to riches and it didn't bother me unduly that my friends who went into jobs on graduation were earning much more than me. It really didn't occur to me to do anything other than do a PhD - I certainly wasn't doing it to postpone making a decision. I actually felt relatively wealthy because my grant I got as a postgrad was far more than the money I had as an undergraduate.

The issue in my case is that I discovered that I didn't actually enjoy doing research for a number of reasons and having a PhD in a non-vocational subject generally makes you less employable rather than more - or at least changes the opportunities on offer. It is also difficult to extract yourself gracefully from a PhD program before the end - I ploughed on to submit my thesis and get my doctorate even though I'd realise a long time before that I didn't intend to pursue a research career. I was also quite naive about the realities of pursing an academic career (despite having a parent who is an academic), but given that research wasn't really for me, that probably wasn't too important in the grand scheme of things.

</comment><comment><USER25> Suppose you have an intellectual interest in some STEM field, and want to learn enough to be able to follow the leading edge of research in that field, and perhaps even advance that yourself. Let's assume you've obtained a bachelor's degree in that field.

Let's consider some options now available to you.

1\. You get a job in programming or IT, perhaps at a startup, and perhaps make a ton of money. You pursue your STEM field interest in your spare time.

2\. You find a job that is actually in your STEM field but that only requires a bachelor's degree. As with option #1 you pursue you advanced interests in your spare time. You probably don't make as much money compared to #1, but you are more likely to do work actually related to your STEM interest.

3\. You spend, say, 7 years getting your PhD.

Let's compare these at the end of 7 years. As far as how much you know about your STEM field goes, option #3 wins by a long shot. With option #3 you've spent 7 years where basically your full time task was to learn your field at an advanced level, interrupted by the occasional undergraduate class you have to teach.

With option #1 or #2, you have something else occupying much of your time. You are going to have progressed in your STEM field nowhere near as far as you would have under option #3. For most people they will _never_ be able to get under #1 or #2 to the point where they would have been at the end of #3...work just doesn't leave enough time. When you are trying to understand the most advanced things in a STEM field you need to be able to devote reasonably long periods of sustained concentration and study.

Financially, those who choose option #1 or #2 are generally better off after 7 years than those who choose #3.

A good PhD program in a STEM field should be fully funded. It will cover your tuition, and give you a stipend and/or teaching assignments that will cover the expenses if a modest scholarly lifestyle.

So the question you have to ask yourself is this: "if someone will cover my expenses so that I can spend 7 years devoted to my STEM interest, getting to a level of knowledge in that field that I probably will _never_ be able to get any other way, is it worth starting the 'make a lot of money/start a family/settle down' phase of my life 7 years later than I could otherwise start it?"

The answer is yes for many people.

</comment><comment><USER26> It's no secret that the life sciences field is over saturated, competitive, and vastly underpaid with a high barrier for entry. Industry demand is low on both bachelors and doctorate level positions.

A bachelors really doesn't get you anything as a sciences student. Besides QA/Tech positions, there aren't many options to cut a viable path within the field. Most companies demand a minimum Masters with a preference for PhD's for most Scientist level positions. Having upward career mobility is very tough without a higher degree.

Spurned on by a lack of better options, most students pursue grad school in hopes of landing a tenure track position or hope their research aligns with relevant industry markets. Competition is insanely high though from the sheer churn of graduate students who are essentially cheap labor for their universities.

</comment><comment><USER27> I did my PhD in theoretical chemistry/biophysics in the late 90s. I was funded throughout so didn't have to take on any debt (I was married for the last 3 years). I learnt a lot during those years including how to really think about a problem, design experiments, and collaborate with the bench scientists on the team. I also fell in love with protein structure, got to understand spectroscopy, the electronic structure of vision, and got to run stuff on some of the biggest supercomputers in the US. PhDs require (IMO) a love of your field, lots of discipline, and tons of patience. I never wanted to be an academic and worked in the scientific industry for 8 years at startups and BigCos (5 of them in product management). The last 8 years I've done something else (although a lot of my interest came from my HPC experience) but have never lost touch with my roots.

Regrets? None. Every experience has been completely worth it.

Many of my friends have done PhDs and postdocs. Some are academics, some lead large research teams, others R&D teams at biotech startups, and some have moved on to different fields like me. Almost none regret doing what they do or the path they've chosen. Sure there is a lot to change with the system and too many PhDs in some fields, but a doctorate was a rewarding experience for me and for a good chunk of those I know who have one.

</comment><comment><USER28> I would never have forgiven myself if I hadn't done a doctorate degree. It turned out that I wasn't a star, I didn't produce the research that would put me on to a tenure-track position with ease. But I don't think there was any way I would have found my limitations without doing it, without seeing how much smarter some people were - I would always have been left wondering what could have been. Now, I have come to terms with it, and moved on.

<comment><USER29> I'm in my 6th year of a (hopefully soon-to-be-completed) doctorate, and my experience has been very similar. I started the degree because I wanted to learn more about a particular field and develop the related technical skills. I couldn't think of a job where I could get the same benefits. I've learned a ton, but have also seen that while I have certain strengths, there are others who are smarter, more motivated, and frankly have different priorities than I do. Like you, I'm now coming to terms with it, but I have a better idea of what strengths I _do_ have and am working towards putting them to good use.

<comment><USER30> Motivation is the big factor. Most PhDs are smart enough to pursue a career in higher ed, but it takes a PhD to evaluate your level of motivation / love of the subject. Intrinsic motivation is becoming even more important given the incentives for entering the field are diminishing (job insecurity, more senior management, publish or perish).

</comment></comment></comment><comment><USER31> One point that has not been made here yet: Because you enjoy doing research and teaching more than you would working in a 9-5 office job!

I'm approaching the end of a 4 year PhD and so far do not miss the world of work too much. But ask me again in a year...

</comment><comment><USER32> What this leaves out is that not all Ph.D. programs are equal. If you are good enough in your field to get accepted into one of the Top 10 graduate programs, you're going to have a great chance of getting a high-quality job when you graduate, assuming you do well. If not, then you're chopped liver.

Those top 10 programs can probably fulfill most of the faculty openings in any given field, nationwide, with enough graduates left over to fill all of the needs of private industry and government labs as well. There is just no reason for recruiters to look further down-list.

I have friends who have taken faculty positions at PhD granting programs ranked 20th or so in the country. They have reported back that their grad students are, um, underwhelming, and they end up doing a lot of their own research hands-on.

</comment><comment><USER33> I'm currently quite happy in an industry job in which my PhD is an asset, but I'd still advise others against pursuing a PhD. I could've found a career path that would be just as fulfilling without a PhD, and I could've gotten there faster if I hadn't spent 10 years on a PhD+postdoc.

My standard advice to people currently in grad school is to drop out with a masters.

</comment><comment><USER34> The article seemingly is very focused on employment. It's taken as axiomatic that people must do whatever they do because it leads them into an employer's arms, at some point.

What about whether a PhD helps people with romantic relationships, sense of belonging in the community, happiness, social ranking, whatever other goals they might have in their lives?

In light of that I think the topic should be changed on HN - I expected a much more balanced analysis. The current headline on the site itself is "The Ever- Tightening Job Market for Ph.D.s".

</comment><comment><USER35> For many it is a calling - a deeply felt desire to explore and seek answers to questions that are currently beyond human understanding. There are few careers outside of academia that enable the focused pursuit of knowledge. Only the most advanced societies in history have supported such endeavors, and public universities were a democratic experiment in extending this support beyond the wealthy elite.

Sadly, it seems that the United States has lost the will and vision to provide this opportunity for its citizens. Now we only talk of job training and ROI. The financial burden has been shifted from the society to the individual, and the true promise of higher education is but a fading memory.

</comment><comment><USER36> I'm actually at the point of considering graduate school to help my chances of establishing a life science based start-up in the future.

Is getting a graduate level degree (at least a Masters) a necessity to be the head of such a company? From what I've seen, most other companies in the space seem to be headed by people with either a PhD or MBA.

<comment><USER37> I dropped out of college to found a med-tech company that went on to be funded by YC. Every other company in my space is founded by a doctor. I haven't noticed a difference in how we are treated yet. If you're good, you're good, and you can always hire the PhDs later on.

<comment><USER23> To be fair, you folks aren't doing research -- you're doing residency program scheduling. And domain expertise is more important for your product than any sort of theoretical knowledge of science or medicine.

To found a company doing scientific research, it helps a great deal not just to have a PhD, but to to be an established researcher in the field you're commercializing. Most scientific startups founded by outsiders are...transparently bad.

<comment><USER37> Agreed. It really depends on what you are going for. You can learn domain knowledge on the fly (a lot of ours is how the organization functions), but you can't gain a deep understanding of the science side like that.

</comment></comment></comment></comment><comment><USER38> My experience: M.Sc. in Computer Science, worked for 5 years in industry, went back to do a Ph.D. but quit after 1 year.

I liked industry much better than doing a Ph.D., even though I loved all the subjects I studied at university.

More details here: [https://henrikwarne.com/2016/03/07/ph-d-or-professional- prog...](https://henrikwarne.com/2016/03/07/ph-d-or-professional-programmer/)

<comment><USER39> I'm sorry but this is the second comment in this thread that is trying to push people to the author's personal blog. I don't think the point of commenting on HN should be to drive traffic.

<comment><USER40> You picked an interesting one to comment on - henrik_w's comments contain, based on a quick scan, a bit more than one link to that blog per comment. That no longer seems like commenting, so much as spamming.

<comment><USER39> Wow I didn't even notice that when I commented. That's like a Michael O. Church level of "hey everybody look what I wrote!"

</comment></comment></comment></comment><comment><USER20> I've just had this debate with several friends (trying to convince them not to go for PhDs), and overwhelmingly their goal is simply to avoid the job market.

Totally anecdotally, it remains very hard to get an entry-level job unless you have some hard skills. A PhD offers the promise of avoiding the grueling process of applying for jobs and they often (incorrectly) think that it will make them much more employable.

<comment><USER41> Apply for jobs is way easier than candidacy and comps, not to mention a dissertation.

I stopped after my masters, bought a house, and had 2 phd candidate friends move in with me (we live in a city with a university). They tried to tell me that grad school was harder than the real world. I argued that its easier citing my experience in grad school and now the real world. I put in a solid 40 hours a week. In grad school, I was at the office roughly 50-60 but it was largely unproductive. I find that graduate labs encourage bad habits like staying up too late to fix a problem that could be easily fixed in the morning after a good night sleep.

To prove to my friends that grad school was easier than the real world, I offered to drive them to university everyday and pick them up on my way home. I noticed that neither of them worked while at home. After the first week of a straight 40 hours, they were exhausted. After a month of this, they found another ride to school because they said working a straight 8 hour everyday was too much!

<comment><USER20> > Apply for jobs is way easier than candidacy and comps, not to mention a > dissertation.

They're different skills.

I certainly would find getting into a PhD problem much harder than getting a job. But I've spent most of my undergraduate career working and networking—and my GPA/research reflects that.

On the other hand, many of my friends never realized that the real world isn't a series of hoops. They've done great academically (3.9+ GPA, serious research with professors) but neglected to develop any real hard skills or to learn about employability. So the job market is incredibly intimidating to them, while a PhD looks like another series of hoops, which is comforting.

> I argued that its easier citing my experience in grad school and now the > real world.

I have no idea what grad school is like, but I do think that the "real world" is a lot easier than undergrad. With jobs, you have a defined amount of work to do each day and have loads of free time to do whatever you want with. Whereas academics have a limitless supply of potential things to do, without defined hours. But maybe that's just because I've always loved work a lot more than school.

</comment></comment></comment><comment><USER42> Two thoughts on why people go for doctorates even though they are unlikely to get a job that requires it.

One is that there is a real conflict of interest for their professors. They are supposed to be advising their students in their best interests, but they need to keep a big surplus of graduate students to keep their graduate programs going, and often also as research assistants.

The other is that American elementary and secondary education is designed to funnel people into higher education, and so that is what students learn how do do, rather than getting a job in the real world. Contrast this with Germany, where there is a very solid track for learning useful skilled manual trades (and that is a key reason German manufacturing continues to be very healthy in the face of Asian competition)

<comment><USER30> Germany has good academic (selective) schools and good technical schools. We used to have something similar in the UK. I have a PhD and was good academically. I recently bumped into an old school friend, who was not academic in the slightest and is now a tradesman earning great money. If I had kids, I wouldn't mind one bit if they decided to pass on higher education and learn a trade instead. I call it, "Revenge on the snobs."

</comment></comment><comment><USER43> There is a very good essay titled "Graduate School in the Humanities - <USER2> Don't Go" that gives a really excellent description of the mindset that exists when young people go to graduate school. This described me well when I was floating around in my early to mid 20s. I enrolled in a doctoral program in industrial engineering and (ended up leaving with a masters) rather than humanities, but still, I see a lot of myself in what this article describes.

[http://chronicle.com/article/Graduate-School-in-the- Huma/448...](http://chronicle.com/article/Graduate-School-in-the-Huma/44846/)

All this is a bit less applicable to some fields. I've heard that employment is good PhDs for CS and some other engineering fields - but then again, so is employment with an MS or BS. Economics may be a good field for a PhD.

One other factor is that getting a graduate degree from a US based university can be a path to immigrate to the US. There really aren't many options for people who don't have family reunification as an option, so going to grad school in a STEM field in particular may have value that isn't captured in pay relative to fields that don't provide this path to the same extent.

The article mentions that enrollment in PhD programs hasn't declined like it has for law schools and historically hasn't been a path to immigration. One way to measure this would be to see if there has been a substantial decline in enrollment in PhD programs from people who already have the right to live in and work in the US.

</comment><comment><USER44> Don't forget immigration. It's a lot easier to get a student visa than a working one, and a lot easier to get a working one with higher degrees.

<comment><USER45> Especially given the current H1B situation. It looks like almost anyone with a PhD and a good immigration lawyer can get an O1, even if they're being hired as an entry-level developer or another position that clearly doesn't require extraordinary skills.

</comment></comment><comment><USER30> Most PhDs pursue their subject because they are very interested in it. That interest may stay the same, it may (in most cases) die down / be satisfied, or it may intensify. You will always be wondering whether academic was for you or not unless you do a PhD.

</comment><comment><USER46> This is a good question to ask. Prospective and current PhD students need to understand the pros and cons, and the incentives driving their recruitment. But in terms of defining the problem, it all depends on your perspective... As a society, we are severely lacking in people doing deep dedicated study in important areas. Understanding biology at a level that allows us to alleviate much of human suffering is still a long way away, for example. So perhaps the question is 'Why do we pauperize people interested in doing serious science and then make them feel bad about it?'

</comment><comment><USER41> I've wondered this for years. I got a nice job at a University Research Lab. I have lots of creative freedom, can write grants, and am well funded. Most of my colleages are Phds yet most of our customers assume I do. I did an honors undergrad with thesis and a masters. I chose a good adviser and after those experiences, I feel I can easily keep up with other phds here but without having to go through the stress of comps, etc.

</comment><comment><USER47> Status, life experience and the opportunity to do interesting jobs that may require additional study.

Why do people keep getting degrees in journalism when there are no jobs?

</comment><comment><USER48> Everything in this article is purely from an american point of view. It's not exactly news worthy that it's a bad system. Also forgotten in there is the journey part of it instead only focusing on it like it was an illogical move in some non-existent game to maximize ones total life earnings. Apart from those pretty big gripes it was an interesting read, just a bit one- dimensional.

</comment><comment><USER49> I did a PhD in pure mathematics decades ago, simply because it was so interesting. None of it seemed practical at the time (eg. category theory, topology, Horn logic) but that changed when computing began to embrace these ideas. In fact, one of my profs suggested that I study whatever I want until I reach 30, when the concrete begins to settle. That was really good advice for me.

</comment><comment><USER50> Serious question: my fiancee is thinking of pursuing a PhD in the humanities field. She has a BA, MA and MPhil, and the natural next step is to get a PhD

She has consistently been among the top 5% of every academic institution she has been in. Her GRE is 333, and she has several research papers under her belt.

Would it make sense for her to get a PhD? Or is the job market too terrible in the humanities?

<comment><USER51> And what would she do with this new title? This should be the main thing that determines whether it's worth it.

<comment><USER52> People focus on the title and what it can buy you, but forget the skill set acquired and what _that_ can buy you.

I have a PhD, but it didn't really help me get my job. What it did do is change me from a worker drone into an independent thinker and designer who can write design docs blindingly fast (compared to my highly esteemed, but slower writing, peers).

Not saying that everyone without a PhD is a worker drone, just that I was.

<comment><USER53> I've known PhDs who wrote very high level design docs full of hand-waved features. When they implemented anything, half of the features were segfaulting and the code base was a mess.

None of this changed their self confidence in the slightest. Curiously, people pushing for formal methods, unit tests and sane design practices were all non- PhDs.

</comment></comment></comment></comment><comment><USER54> The 40% with employment commitments doesn't seem that bad actually. It's probably higher than the percentage for B.S. graduates.

</comment><comment><USER55> 3 most important motivations are:

1) Love for science (it is hard to do full-time funded research without a PhD)

2) Prestige

3) Desire to work for the government (some high-profile public service positions make it easier to get in with a PhD).

Nearly nobody is doing a PhD for "employability" (except specific target positions from #3) or financial reward.

</comment><comment><USER56> i'm a drop out so i may be biased, but my experience of software developers with doctorates is that they are very consistently not as good as self taught juniors in the workplace.

this makes sense.. a doctorate is a research effort with a very different kind of end goal to the typical workplace environment. its an opportunity to explore the field and do work that most employers will never let you - <USER3> more importantly the skill set required is, imo, highly orthogonal to the skills require for day-to-day 'boring' work.

why do people expect doctorates to result in employment outside of academia? they are poorly suited to this by design... and afaik its completely intentional because of the aforementioned difference in skill requirements.

<comment><USER57> There's another selection bias at play: the really good PhD's in CS who didn't go to academia and aren't doing research at a research lab, are making millions at Google/Facebook/quant finance, and not <foo corp>.

<comment><USER56> its funny how all of the best programmers i know have quite a lot of disdain for the idea that Google or Facebook are good at engineering... their public facing code is good evidence of it too. HHVM is full of rank amatuer crappiness, as well as some fairly cool stuff... but mostly crappiness and WebRTC is an absolute nightmare to use in practice compared to most rookie efforts in the same area (and its not because of what it does, its because Google have lots of internal stuff to work around the shittiness that isn't public afaik).

i've met plenty of very shitty googlers and facebookers, as well as some truly excellent ones. but i suspect you will find much better guys in games, vfx, military and finance (which at least backs up some of your point). in my experience there is a much lower tolerance for crappiness in those fields by necessity. the performance and quality bar they have to meet is orders of magnitude higher in a lot of cases...

</comment></comment></comment><comment><USER58> Everything explained here. [https://www.ted.com/talks/ken_robinson_says_schools_kill_cre...](https://www.ted.com/talks/ken_robinson_says_schools_kill_creativity)

</comment><comment><USER59> I think the answer is very simple - a doctorate still packs a lot of prestige. It says that one is Smart, especially nowadays when more people than ever hold a bachelor's.

</comment><comment><USER60> Another point is that advanced degree in the right subject make acquiring a work visa easier in many countries, though US work visas are challenging to get regardless.

</comment><comment><USER61> Well, I was paid for the first five years of my PhD... and then I dropped off :)

</comment><comment><USER62> Because a lot of employers of researchers demand it.

</comment><comment><USER63> 1\. Decent Scholarship

2\. Better long term prospects

</comment><comment><USER64> This article is clearly written by someone without much perspective into the actual people getting these degrees, the programs that offer them, or the process by which they are granted. But why let that stop you from writing an article based on some numbers you saw?

First of all, yes, the market for academic jobs in general is pretty bad, despite increasing demand for college education, because of relentless government slashing of public higher education funding over the past 20 years. The comparison to what has happened recently with law schools is completely senseless, though. Law school was seen by many to be a certain path to an extremely high-paying job, and was pursued by the most ambitious, most money- driven individuals. And for a long time, the reward really was there. That all changed suddenly, and the jobs dried up, so when most of the reason applicants choose a law career goes away, of course applications will drop.

But many academic fields have _always_ had poor job markets, few tenure-track positions, and cruddy pay. It may be getting marginally worse every year, but it's not like there's a sudden sea change in what you can expect if you go get a degree in English. And in any case, the sort of person who pursues a PhD is almost never motivated by a desire for money, or a "great career". The people who are so motivated are the ones who look at tables of graduate school ROI and used to choose law. Your Art History PhD student probably has never seen such a chart, wouldn't care if they did, and doesn't think in terms of ROI.

The other problem with the law school comparison is just raw numbers. Law schools likely put out an order of magnitude more JDs each year than all typical PhD fields combined. The PhD pipeline is far more stable.

The author questions why schools are pushing more people through graduate programs when demand is decreasing, but that's easy to answer. First of all, at the micro level, each individual department and PhD program has an internal incentive to continue the work they've been doing for years. This is their job, after all, and there are customers.

The universities that run these programs also have every incentive for them to continue. 1) Accreditation boards, US News and World Report rankings, and research funding all give you bonus points for having a PhD program. 2) Graduate tuition is higher than undergraduate tuition. 3) PhD students are an even cheaper source of labor with even fewer rules about mistreating them than adjunct faculty! This one is huge. There are some schools where most of your classes will be taught by PhD students, not by actual professors.

The author also doesn't seem to have ever talked to anyone who's gotten a PhD. Those median time-to-completion numbers are meaningless. Yes, some fields (mostly hard science) have their PhD candidates doing research for their dissertation for years on end, but in most fields, you do a couple of years of PhD coursework and then you work on your dissertation until you finish. Many programs will pay you a tiny bit to stick around if you will teach classes, at least for a couple more years, but in general the onus to complete this massive project is on the individual student. And given that they probably have families by this point in their life and that they're most likely working some unrelated or semi-related job in the meantime to pay the bills, yeah, sometimes it takes years and years to complete the dissertation.

Go to any academic-related administrative department at a large state university and you will find it filled with people who have full time jobs supporting the university, but are also slowly working through their PhD in Education or something similar. They get a slight bump in pay and respect when they complete it, but for the most part they are there working for the university and working on their PhD because they love their field, and they're passionate about helping students learn.

Of course, the stats given also leave out the fields in which there is plenty of demand for PhD-holding professors. If you want a six figure starting salary and schools falling over themselves to hire you, go get a PhD in Accounting.

Finally, there's not enough context in the article on the stats showing "no job commitment" for PhD students. It apparently means "tenure track professorship", but I think you'll find that most of the people in those fields know exactly what they're getting into, end up in totally acceptable jobs, and are pleased about doing the work to get their PhD.</comment>


Show HN: My Django book, Hello Web App, is now free online - <AUTHOR> A couple of years ago I wrote Hello Web App (http://hellowebapp.com) to teach beginner Django web app development. It's aimed at non-programmers and designers, and walks folks through building and deploying a basic web app.
I'm working on my third book to teach web design fundamentals to programmers (Hello Web Design: https://www.kickstarter.com/projects/1868398473/hello-web-design-design-basics-for-non-designers), so I've made Hello Web App free online during the Kickstarter campaign for the new book.
The full Hello Web App tutorial can be accessed here: http://hellowebapp.com/tutorial/intro/
Also, I'm happy to answer any questions about writing and self-publishing books — I've probably made about $20,000 lifetime profit on the two Hello Web App books overall.
Thanks all! <comment><USER2> As a programmer who has a desire to build things that require UI components, but no real desire to learn the intricacies of UI, what I personally want is a cookbook/toolkit that lets me do common things really easily and then make tweaks, not build anything from scratch.

I really liked Bootstrap for this for the web, particularly things like their dashboard templates which gives me a cookie cutter dashboard with minimal effort.

<comment><AUTHOR> I think you'll like Hello Web Design then, since my number one push is for shortcuts and easy guidelines to remember, then teach a bit about the theory and fundamentals (usually taught the opposite way and shortcuts are usually not taught at all).

Check out my article, which gives a little preview of the content: [https://medium.com/@limedaring/design-for-non-designers- part...](https://medium.com/@limedaring/design-for-non-designers- part-1-6559ed93ff91#.5u8g9ob0u)

</comment></comment><comment><USER3> I saw you speak at CUSEC, cool stuff!

<comment><AUTHOR> Aw yay! Super proud of that talk, it's my favorite. :)</comment></comment>


Secret Codes and Features in Samsung Galaxy S3  - <AUTHOR> http://techpp.com/2012/07/31/secret-codes-features-samsung-galaxy-s3/

<comment><USER2> Nice one</comment>


Zone2, the time tracker for Mac - <AUTHOR> http://rinik.net/zone/

<comment><USER2> I just bought this app, and pretty much immediately gave it a five star written review on the app store. I have been desperately searching for a tool that allow me to be able to quickly switch between projects, record irregular intervals of work for a given project, and export that info as some sort of document (CSV preferred). This app meets that need with a great UX, and without all the unnecessary fluff of expense tracking/project management that others seem throw in.

</comment><comment><USER3> Is there a way to archive projects? I would like to use this for my work, but I wouldn't want to have a longs list of past projects in the drop-down.

Also, there aren't tasks, right? Sometimes it would be nice to have several tasks under a project.

</comment><comment><USER3> One thing about things going into the Mac App Store is that there is no trials. This looks like it may be great for my needs but there's no way to try before I buy... now, it is just $3 so, why not?

<comment><AUTHOR> Hey why don't you email me at firstname.lastname@example.com and I'll send you a free promo code.

For me it's worth giving a few copies away to users who like to give feedback. :-)

About archiving, I've been asked about this before, and now I'm pretty sure this will be needed for the next version. So maybe you are better off waiting for the next release.

<comment><USER4> I'm a freelancer, and currently I'm using the free Time Tracker.app (<https://www.macupdate.com/app/mac/20310/time-tracker>). Yup, this looks clean and simple. i wished there's a free trial somewhere (although not sure if it does the same thing). Mind giving a video of the app flow if you don't have any more promo codes? :)</comment></comment></comment>


Relativity by Albert Einstein (Book) - <AUTHOR> http://books.google.cl/books?id=O5f3L2GfXBQC&lpg=PP1&pg=PT1

<comment><USER2> What about it?

<comment><USER3> It's reputed to be _very_ good.</comment></comment>


Show HN: A J Cheat Sheet - <AUTHOR> https://github.com/dankilman/j-cheat-sheet/blob/master/cheat-sheet.ijs <comment><AUTHOR> I've recently finished a book on the J programming language (Learning J) and summarised what I read in what turned out to be a rather useful (at least to me) cheat sheet.

Other similar content I found was not very "ctrl+f"-able, this one is.

Perhaps someone else will find it useful as well.</comment>


Postgres.app: the easiest way to run PostgreSQL on the Mac - <AUTHOR> http://postgresapp.com/

<comment><USER2> `brew install postgres` is a close runner up (note: requires homebrew.)

<comment><USER3> That method doesn't give you a running Postgres; you still need to configure the cluster yourself.

</comment></comment><comment><USER4> No Snow Leopard support?</comment>


Why doesn't Quora use PostgreSQL - <AUTHOR> http://www.quora.com/Why-doesn%27t-Quora-use-PostgreSQL#ans72931

<comment><USER5> This is a common mistake people make, I think. Just because an ecosystem is smaller does _not_ mean it is inferior (or less practical). In fact, the opposite is routinely true.

Is it harder to find tutorials for PostgreSQL? Yeah, maybe. Is it harder to get high-quality advice and resources from subject matter experts? No; in fact, it's far easier. I have used PostgreSQL for many years and in all of them every question I've had has been answered quickly and properly by somebody in #postgres who knows way more about postgres and SQL than I'd ever even _want_ to know.

And what about hires? When a million people "know" MySQL it takes a lot more effort to find somebody who actually _does_. There may be fewer contractors for PostgreSQL but they are practically by definition better than the average MySQL contractor.

 _unlike learning a programming language, with databases, the domain knowledge is harder to discover and more risky if you don't understand it_

Exactly. So should you trust the ecosystem that relies on lazy, bandwagon developers for the majority of its market share, or the one that has been silently doing shit the right way for years? There may be a whole lot of really skilled MySQL admins out there (I guess there must be given how many huge sites go to extreme lengths to make MySQL work for them), but I can find you a handful of grossly overqualified PostgreSQL admins in no time. The same can't be said for the former.

<comment><USER6> That's really interesting to know, actually. I've only been with companies that had MySQL installations (Facebook and Twitter) and so jumping in was really easy, but on the other hand I do like to do things right. A few questions, though:

0/ How would I be able to tell if the limitations of our DB are because of MySQL and would be solved by PostgreSQL? That is, common pain points in MySQL that we wouldn't recognize immediately until we switched? (Edit: mostly looking for performance-related, but anything is fine.)

1/ Is there a PostgreSQL consulting firm that has the same reputation that Percona does with MySQL?

2/ Is there a good place to start when there's an issue or question I have with PostgreSQL (aside from IRC)? With MySQL I've been either going to the docs, or to Google, but it's been spotty with either.

<comment><USER7> 0: Empact lists a few here: <http://news.ycombinator.com/item?id=1547192>

1: There are a number of highly trusted PostgreSQL consulting firms; the last time my employer needed a consultant, we hired one from Fujitsu's dedicated PostgreSQL team.

2: So far, I've only needed the (deliciously comprehensive) PostgreSQL online docs and (sometimes) Google; most Google searches wind up on the official user/developer mailing-lists, so that would be another good place to ask.

</comment></comment></comment><comment><USER8> Ok. Being borderline trollish here, but if I read this article, this means to me "yeah. we are using a technically inferior solution and thus risking data loss and increasing the workload on our developers because our HR department gets more resumes if we search for an in-house database specialist".

Postgres usually does its thing and does it well. You usually don't get weirdness that requires you to even hire a specialist. And if you do, the consultants around are great and actually understand the problem.

As other people here have said: If you find someone listing PostgreSQL experience on their resume, you know they have the experience. If you find somebody listing MySQL, you'd have to check whether they are just listing it because they have heard of it or whether they really understand the problem domain.

So. DB specialists are settled. What about developers? PostgreSQL and MySQL are close enough in what SQL constructs they support (no wonder - <USER2> provides a subset of SQL anyways), so coming from MySQL to PostgreSQL usually is trivial and the few things that are unsupported unix_timestamp() for example, you can write wrappers for if you really need them.

Of course, developers will at first find it strange that they can't, for example insert a 200 characters string into a varchar(10) column, but even the most braindead developer sooner or later will understand that silently truncating data usually is a bad thing.

No. The arguments listed in that article are just fear of the unknown and total non-issues (in german, there's this saying: "Was der Bauer nicht kennt, das frisst er nicht" - ever so true).

I'm glad I can work with PostgreSQL day-in, day-out. I'm glad I migrated that huge application from MySQL to Postgres back in 2000. I'm glad I did not touch MySQL for serious stuff ever since then.

I understand if you need to use MySQL if you want to run some software that runs only on MySQL. But if you are writing your own application, there's just no reason not to got with the real RDBMS.

/rant.

<comment><USER9>
      I'm glad I migrated that huge application from MySQL to
      Postgres back in 2000. I'm glad I did not touch MySQL for
      serious stuff ever since then.

You should refrain from commenting on MySQL then. It's amazing (and sad) how many pro-PG-anti-MySQL folks don't even know basic things about MySQL but feel entitled to rant and call it "toy DB", "not real RDBMS", etc.

<comment><USER8> note the emphasis on "serious stuff".

I did in fact touch MySQL since then in the process of setting up various open source applications that require MySQL.

While back then, I needed Postgres for some features MySQL didn't have but otherwise didn't really dislike MySQL, over those 10 years, I really learned to hate MySQL:

\- try (erroneously) adding or inserting latin1-data into a UTF-8 table. You'd expect an error message, but instead your data gets truncated at the first byte with the high bit set. No warning. This means that you have to re-read and compare whatever data you stored to be sure that you didn't lose any.

\- once, mysqldump stopped dumping after coming across a corrupt table. It did so while still exiting with a 0 exit code. That meant that for months my backups were incomplete.

\- "alter table sometable add somcolumn integer not null" will gladly add the column and place null values in each row. Null values that should absolutely not be there. No error. No warning.

\- Same goes for date columns, but there it's even better as it'll set the date to 0000-00-00 which is invalid if you try to add this in an insert or update.

\- I said "corrupt table" earlier. Yeah. That happens with MySQL, whereas it never ever (with the exception of faulty hardware of course) happens with any other RDBMS - including PostgreSQL.

Now, in my opinion, a database that doesn't guarantee that the data you put in can then be read back out in unaltered form (or at all) is, really, no real RDBMS.

Sorry.

(edit: reason for not posting this in my original reply: I'm basically repeating above points over and over here on HN and I thought people would know my gripes by now)

<comment><USER9> So basically you learned to hate MySQL while not doing anything serious with it. Interesting.

      - try (erroneously) adding or inserting latin1-data into a UTF-8
      table. You'd expect an error message, but instead your data gets
      truncated at the first byte with the high bit set.

False.

      - "alter table sometable add somcolumn integer not null"
      will gladly add the column and place null values in each
      row. Null values that should absolutely not be there. No
      error. No warning.

False.

      Same goes for date columns, but there it's even better as it'll
      set the date to 0000-00-00 which is invalid if you try to add this
      in an insert or update.

That's why I said those who don't know MySQL like to complain — hardly any of doing the rants have heard about sql modes:

      mysql> alter table t1 add column (dob date not null);
      ERROR 1292 (22007): Incorrect date value: '0000-00-00' for
      column 'dob' at row 1

      - I said "corrupt table" earlier. Yeah. That happens with
      MySQL, whereas it never ever (with the exception of faulty
      hardware of  course) happens with any other RDBMS - <USER3>
      PostgreSQL.

Funny. Googling with right keywords paints a bit different picture. Must be some other PG, I guess.

<comment><USER8>
              - try (erroneously) adding or inserting latin1-data into a UTF-8
              table. You'd expect an error message, but instead your data gets
              truncated at the first byte with the high bit set.

        False.

<http://www.gnegg.ch/2008/02/failing-silently-is-bad/>

            - "alter table sometable add somcolumn integer not null"
            will gladly add the column and place null values in each
            row. Null values that should absolutely not be there. No
            error. No warning.

        False.

ok. it inserts them as 0 - that's in improvement from last I checked. Still. It silently does stuff. It should not let me do it in the first place.

Same goes for the date. And please don't tell me the error message you get back there is quite correct. That alter table statement tries to silently add a default and later fails due to the default not being valid.

Also, see this:

        mysql> alter table foo add dob date not null;
        Query OK, 0 rows affected (0.00 sec)
        Records: 0  Duplicates: 0  Warnings: 0

        mysql> describe foo;
        +</comment></comment></comment></comment></comment><comment>----+</comment><comment></comment><comment>-+</comment><comment>+-----+</comment><comment>---+</comment><comment></comment><comment>----+
        | Field    | Type        | Null | Key | Default | Extra          |
        +</comment><comment>----+</comment><comment></comment><comment>-+</comment><comment>+-----+</comment><comment>---+</comment><comment></comment><comment>----+
        | id       | int(11)     | NO   | PRI | NULL    | auto_increment |
        | value    | varchar(10) | NO   |     |         |                |
        | intvalue | int(11)     | NO   |     | NULL    |                |
        | dob      | date        | NO   |     | NULL    |                |
        +</comment><comment>----+</comment><comment></comment><comment>-+</comment><comment>+-----+</comment><comment>---+</comment><comment></comment><comment>----+
        4 rows in set (0.01 sec)

        mysql>

this table now contains two columns that can't be null and yet the default is null. Thanks.

Concerning corrupted tables. I have around 1.5 TB of data spread over ~10 databases and over the last 10 years, I never had a corrupt table that I could not trace back to hardware failure (harddrives, raid controllers, RAM).

Over the same period, I had around 200 MB of data spread over ~10 databases in MySQL and I had three cases of corrupt tables that I could not trace back to defective hardware as various diagnostic tools yielded no error and the machines still run without problems with PostgreSQL.

Maybe I'm just unlucky.

<comment><USER9> Maybe you just don't know what you are doing.

<comment><USER10> Maybe you just have nothing to say and don't recognize that such behavirs are clearly wrong, that failing silently for a database is a case of bad design, and MySQL is just technically inferior to Postgres. Or are we missing something?

<comment><USER9> Yes you are. The first thing you are missing that such behaviors are configurable: you can choose whether to fail silently or get an error. The second thing you are missing is the reason such behavior exist in the first place — namely, there are usage scenarios when it is preferable. Having ability to choose which trade-offs to have is in no way technical inferiority.

<comment><USER10> Never in my entire experience as a software developer have I seen a case where a database should fail silently. It's bullshit and you know it.

</comment></comment></comment></comment></comment><comment><USER11> A few bugs/limitations of Mysql not shared by Postgres. All AFAIK.

* Integer columns with null constraints will silently translate nulls to 0s, rather than throwing an error (as constraints in all other contexts do). Imagine a database which, on violation of a foreign key constraint, just silently picked the first row in the target table - it's just insane.

* Transactions can't roll back schema changes, so if you have a Rails migration which fails on the application of a unique index in the midst of other operations, too bad. Your DB is stuck in the no-mans-land between migrations.

* Constraints are anemic relative to postgres: no integer range or string length constraints, no whitelist, blacklist, regex match, &c. &c.

There are certainly others I've run into, but the above indicate to me that the Postgres developers are working with/delivering a better code-base.

<comment><USER12> > Transactions can't roll back schema changes

That's a feature of postgres rather than a limitation of mysql however. As far as I know the SQL Standard doesn't ask for transactional DDL, Oracle has pretty much no support for transactional DDL (in 10g anyway, each DDL statement is executed in its own transaction and any pending transaction is COMMITted before a DDL), I have no idea about SQL Server.

<comment><USER12> (and don't get me wrong, it's a totally awesome feature and it should be praised to high heaven, but it's an additional feature nonetheless, rather than a limitation of the concurrence)

</comment></comment></comment><comment><USER13> Is quora flooding their own site with self-posed questions trying to get publicity and rankings? The only time I've ever heard of it is from the oxygen of publicity HN seems to give it.

<comment><USER6> Nah. You can usually (but not always) tell who asks the question by checking the first person who's following the question, or by checking the question history by clicking on the timestamp.

</comment></comment><comment><USER14> It's interesting to see how PostgresSQL didn't get nowhere near the popularity of MySQL. I think a few major things contributed to this:

1) MySQL focused on performance first while PostgresSQL positioned themselves as "feature-complete."

Of course, that was years ago and in the meantime they became comparable in both aspects. MySQL+InnoDB is ACID compliant and Postgres matches the InnoDB performance.

Sure, Postgres might still have the upper hand in the variety of features but MySQL has MyISAM. If you don't need transactions you can get a significant performance boost over Postgres. Speed IS a feature and one could argue it's more important to many than all the nice extra things Postgres has.

2) MySQL was compatible with mSQL so they got a pretty good user base right from the start. mSQL later became obsolete.

3) Inertia. Even if Postgres would be net superior to MySQL (think feature complete but faster than MyISAM) it would still take a lot of time to catch up, if ever. There are tons of tools already developed for MySQL and a huge knowledge base around it. Just like C++, Windows, or any other big technology, there's a lot of inertia to it.

<comment><USER15> I think having a "launch" at the right time might have had some impact also. MySQL launched in 1995, right around the time that the idea of generating hypertext dynamically was taking off, so a bunch of web stuff was looking for a lightweight, fast DB without necessarily a lot of features. Postgres had been around since the 1980s as a research project, and in 1994 was publicly released as an open-source project and added SQL support (rather than its own query language), but it was sort of off the internet's radar as a research project mostly known in academia. The official launch of the rebranded PostgreSQL was in 1997, but that might've been a bit late already, since 1995-1997 were a fairly crucial two years for the LAMP stack's birth.

<comment><USER16> Most people don't realize how bad postgress was at that time.

I tried postgress in 1995-96. I was tiered of mSQL [M-SQL not MySQL] and really wanted a free SQL server to work at backend of our web apps. Downloaded, compiled and installed what was then called postgress95 on our linux server. And then I could connect to it from localhost using the command prompt. But the instant I connected with it from a remote machine over TCP/IP the server crashed! I really wanted to make it work, couldn't. Note that Internet was still very new and support availability wasn't as wide as today.

Discovered and started using mySQL which was then way better than mSQL, And still using it after 15 years.

</comment></comment></comment><comment><USER17> error 500. maybe its unrelated to them not using postgresql.

<comment><AUTHOR> here's the correct link: <http://www.quora.com/Why-doesnt-Quora-use- PostgreSQL>

HN must have stripped out the ' from the link when I submitted it

<comment><AUTHOR> grrrrr. apparently HN always strips single-quotes from links. There should be single-quote in the link in doesn't.

Here's a bit.ly link :-( <http://bit.ly/9iQWn8>

<comment><USER18> You could always URL encode it: <http://www.quora.com/Why-doesn%27t-Quora-use- PostgreSQL>

That way everyone clicking on the link from the home page doesn't get taken to the 500 page.

</comment></comment></comment></comment><comment><USER19> There are a number of companies that provide support for PostgreSQL. Enterprise DB is probably the most prominent of them. Kind of funny that they didn't have a good technical reason not to use PostgreSQL, just that the community was smaller.

Postgres 9, with streaming replication and easy to setup bullet proof read only slave replication will probably move momentum in the direction of PostgreSQL. The replication was always easier to setup on MySQL, which gave it an advantage.

</comment><comment><USER20> Excellent lesson to other startups: When two or more products do what you need, choose the one you can more easily hire help for.

(not that there aren't a fair number of postgres experts, but there's no question there are far more competent mysql experts)

<comment><USER12> No question, really?

</comment></comment><comment><USER21> This article is an example of choosing easy option over the better option. From a startup's perspective, it's completely understandable. Thousands of cheap MySQL admins, or a few hundred expensive PostgreSQL admins?

As for the "nobody uses PostgreSQL" argument; Facebook and Twitter aren't an endorsement of MySQL because any Linux hosting provider out there will have MySQL installed BY DEFAULT. PostgreSQL would require you to upgrade to a more expensive account (for root access) and install it yourself. As a result, all the hobbyists learn on the free and always available MySQL when building "The Next Big Thing." Remember, Facebook was started in a college dorm room.

</comment><comment><USER22> Those who commented in favor of postgres: What are some of the biggest deployments of postgres? Facebook has a gargantic deployment of MySQL and it just works. The author of the answer on Quora mentions reliable scalability as the most important factor for their MySQL choice on another answer here: <http://bit.ly/dm6HtQ>. What is the biggest system that postgres is deployed in?

<comment><USER23> A better question would be: who uses MySQL as anything but a glorified key- value store when reaching large sizes? I personally have been involved in trying to use MySQL as a "real database" in the 10-100TB range and let me tell you, it's not pretty. I'm not sure about the open source PostgreSQL, but I know Greenplum has petabyte level warehouses running on a distributed version of it.

A few observations:

1) MyISAM's performance is highly dependent on certain idiosyncrasies of a lot of applications. Using MyISAM in this day and age is a very bad idea. InnoDB at least gets closer to real database behavior.

2) The "query optimizer" is insulting at best and actively impeding getting things right if you use it for much more than simple queries. Something that's more along the lines of what really large databases (as opposed to KV stores) get used for can implode the server.

Personally, I think too many people try and stick things in relational databases that don't belong there simply because they've got the hammer in their hand and it's easier than pulling out a screwdriver.

<comment><USER22> _who uses MySQL as anything but a glorified key-value store when reaching large sizes?_

My assumption is both Quora and Facebook use MySQL this way. While you are right that this is not using it as a _real database_ , I want to know if PostgreSQL is deployed in a similar setup at all. Most of the people (including you) don't take into account the fact that there are many cases where MySQL (used as a KV store) proved to work, while I have never heard of such huge PostgreSQL deployments. If this was a general discussion regarding MySQL and PostgreSQL I could understand that, however, I think the post is more about whether to choose MySQL or PostgreSQL if you are going to use it as a KV store.

<comment><USER23> The introduction of hstore actually lets you do this natively. I think it's partially that PostgreSQL people tend not to try and use the hammer as a screw driver, but maybe that's just me.

Or maybe sometimes MySQL is a screw driver being used as a hammer?

</comment></comment></comment></comment><comment><USER24> That's why database abstraction is so nice. If I ever get sick of MySQL - <USER4> say, decided I didn't want to use Postgres - it's not hard to switch!

<comment><USER14> Database abstraction isn't, or at least isn't practical.

Small projects don't need an interchangeable database. Bigger projects won't let you get away without making assumptions about the database. Nothing but the basics work the same between SQLite and ORACLE.

<comment><USER24> SQLite... Oracle? never used either one, but I believe it.

I'm talking about MySQL vs. Postgres.

</comment></comment></comment><comment><USER25> there's one useful thing about postgresql:

specifically, interviewees that inform you you should be using postgresql can be immediately rejected.

"do you use mysql or postgresql?" "oh, mys-" "YOU SHOULD SWITCH TO POSTGRESQL!!"

<comment><USER26> I think this can be extended to any interviewee who starts redesigning your stack before they're even hired can be rejected.

<comment><USER8> I don't know. Personally, I'd prefer an interviewee that comes to the interview with an open mind and actually tries to make suggestions based on the view of the infrastructure that they have acquired based on the information I have given them or they inferred in their research of my company.

Of course they might be wrong because naturally, they can't have the full picture. But I'd much rather have an employee I can hold a discussion with and get input from then one that blindly does what I tell them to. In the end, I might as well have made a bad decision or overlooked something.

In fact, having an engaging discussion during the interview is quite a sure- fire way to get hired if you are in an interview with me (provided the stuff you talk about makes sense).</comment></comment></comment>