Ask HN: Late twenties, last four years spent fiddling w/projects. Employability? - <AUTHOR> Hi HN! I'm a 29-year-old programmer who left my SV job four-odd years ago to pursue a life of travel and indie development (at least until my savings ran out). Travel-wise, things have just been peachy. Career-wise, I've accomplished a whole lot compared to what I was doing before, including developing a bunch of small/medium-sized native iOS and OSX apps (one of which grossed $10k+), publishing a (simple == sub 1k line) open source project that has close to a thousand stars, blogging about technical topics (nothing too fancy), and implementing/blogging a very interesting CS research paper. If I play my cards right, I have maybe 1-2 years of independent living left, assuming I don't get another "hit".
Still, I am filled with dread. By my own evaluation, my progress has slow, and I feel like I could have accomplished 2-3x as much during my "sabbatical" if only I spent a bit less time procrastinating and forced myself into unfamiliar programming territory. (I perceive this as having wasted about 2-2.5 whole years out of this period.) I have a recurring, deep-seated fear that if my cash runs out and I have to re-enter the job market, I'll simply find myself unemployable. Although I'm almost 30, I don't really view myself as a senior engineer material; I can implement stuff just fine if given a blueprint, but my experience with high-level architectures and complicated, stateful systems is lacking. (Working on getting better at this, though. Lots of projects in the pipeline.) I also have zero managerial experience since I've been entirely working by myself. On the plus side, I have a CS degree from an excellent university, though I imagine the importance of this wanes as you get older.
HN, are my fears justified? Am I just falling into a bottomless pit, career-wise? Or is this stuff correctable? <comment><USER2> No, should you ever want a job again, on your CV put that time down as a company of your own creation and list it like you would any other job. What you made, what you achieved, etc. I did this about 4/5 years back, didn't have any problems getting interviews.

When you start getting interviewed they'll ask why you're leaving self- employment and all you have to say is you tried running your own business, it didn't work out, and you want the stability of a full-time job again. That'll be about the extent of it.

References are a little trickier, I had some consulting clients that I'd done part-time work with, so had people to ask. To be honest, references are at the "we want you" stage so they're a bit more flexible.

The ultimate problem I found was being an employee is frustrating so I left full time again and switched to consulting instead.

</comment><comment><USER3> The only thing that you mentioned that would make me a little bit nervous is that you have been working entirely by yourself.

I would think a github repo with 1000 stars would give you the opportunity to work with others. Maybe start accepting some pull requests, or join another open source project.

Otherwise, everything else that you mentioned seems like pretty run of the mill imposters syndrome. You built some products, and real users ended up using these products. That seems like some pretty solid experience right there.

</comment><comment><USER4> I would also think about how you present yourself. Some things to consider:

1\. Would you want a contract gig or a salary position. If later, be sure to make sure you don't sound like a drifter. Employers don't want to invest time in someone that doesn't plan to stay around at least a little while. 2\. Can you do some contract gigs now to keep up skills and resume? 3\. As for developing your own "hit", I think the idea is more important than the tech stack

</comment><comment><USER5> Keep doing it - just take on some contracting work to extend your runway. ios contracting work should be really easy to find.

</comment><comment><USER6> Don't give up now. The risk is good for you. Use it to "light a fire under your ass," as the saying goes.

Be selective about opportunities and don't underestimate the advantage of your position. Many people spend years employed dreaming about the sort of independence you've achieved.

</comment><comment><USER7> What was your programming level when you left the job and started working on these projects?

I view myself as probably a 4/10 at this point, really never having built a super complex application from start to finish. And that's the fear holding me back. However I feel it would light a big fire under me to improve in a lot of ways, be super efficient with my time, study hard and put it into practice. Any advice about this "imposter syndrome" feeling would be appreciated.

</comment><comment><USER8> Your employability is also going to be dependent on if you're willing to relocate and what you are willing to accept for a salary. I think you should land fine as long as you've got decent "emotional intelligence"/teamwork/sociability skills and you are willing to accept starting off at place that isn't necessarily perfect for you, but good enough.

Experience is what it all boils down to, and it sounds like you've got nice set of experiences to draw from which might set you apart from other candidates.

</comment><comment><USER9> Maybe obvious, but so much seems to be a confluence of factors beyond your control and ones within your control. Luck plus making the right connection at the right time would help. Also doing well in an interview is a no brainer. Could you make another indie app & gross more money and live cheaply? Bosses kinda suck imo (but maybe might be cool sometimes for a while) so running my own show is always an appealing possibility.

</comment><comment><USER10> Firstly your experience is very good, and it must be very enriching. Do not undersell yourself, you can give a very positive spin on your indie experience. You can cite this as your entrepreneurial foray.

Secondly, why not try for few interviews at good companies, and see how you like the feel - You might like it - If it does not work out well, you can always go back.

</comment><comment><USER11> It seems to me that your fears are unjustified. Being unemployable as a senior, architect level developer is very different from being unemployable as a developer, period. It seems to me that you have the skills and resume to find a senior role that is not that senior as architect or CTO. Or at least a more junior one with real fast growth.

</comment><comment><USER12> Off-topic, but I've been thinking about doing my own sabbatical in order to focus on my own projects and hopefully start a business. Can you share anything about what you did to get started? How much money you had saved? How did you adjust your life?

<comment><USER5> Not OP, but I did this and it really depends on what you want.

I had some success, then got obsessed with AI. After 1-2 years studying nonstop figuring out what to build I ran out of money and had to consult.

I would have made more money working full time... but perhaps not if my next project goes well.

That said, even with what I know now, I'm about to do it all again. The freedom is priceless.

Risk is scary. But going 9-5 to a job hoping for raises, good bosses, and stock liquidation events is even scarier IMO.

</comment></comment><comment><USER13> You're not unemployable and don't underestimate the value of your experiences over the last four years.

</comment><comment><USER14> I feel you bro...</comment>


Ask HN: What should I learn next? - <AUTHOR> For the better part of the last 2 years I've been freelancing part-time as a Wordpress developer, focused mainly on building custom themes and modifying existing plugins. Naturally, this meant PHP was the thing to learn, since I was already familiar with HTML/CSS, and 99% of the time, a bit of jQuery on the side was just enough to get the job done. However, after taking a break for a couple of months in order to focus solely on school, I've realised something pretty scary: I actually suck at programming. What I mean by this, is not that I can't code ( I've pretty much learned programming by myself and am passionate about it ), but the fact that this entire time I've been half-assing it, and now I find myself in a place where I'm not really good at anything. I mean, I know a little bit of C that I picked up in school ( although no OOP ), enough PHP in order to be able to write simple functions that hook into Wordpress, a little bit of Ruby I picked up while building a Rails app, and although I am pretty good with HTML/CSS I'm definitely not up to date with the latest technologies.
So now, after graduating high school and making the decision to get into full-time freelancing, I'm faced with the following dilemma: What would be the the best thing to learn next? I am interested in web development ( although it's not excluded I'm going to learn iOS dev. too sometime in the future ), and after a lot of research I still have no idea what's the right way to go. Foe example, I know I could get serious about learning Rails ( or Django, for that matter ), but then from what I've been reading more recently, javascript ( and javascript-based MVC frameworks ) are the next big thing, and that's what I should focus on.
I know this is a pretty subjective matter, but I'm hoping there are some general guidelines which will help me make the best decision. Thank you! <comment><USER3> Postgres. Get a book by Joe Celko (SQL for Smarties, for instance) and really learn relational databases. They're not going anywhere. Any job you take, you can bet there's a relational database back there somewhere. Postgres is free and will surprise you less than any other. If you're very good at SQL, you will write a lot less code in the middle tier to post-process data on the way out. You won't be hindered by a lack of OO understanding--in fact, it may benefit you since you won't face the O/R impedance mismatch. Once you're good at SQL you'll be a lot less worried about other things in the backend, and the declarative nature will help you reason better. It will pay dividends no matter what step you take after that.

<comment><USER4> I'm going to agree, as a Sr Database Architect, that once you have these skills and some experience under your belt you will never want for jobs.

There is one huge caveat. Sometimes it is so boring that I often daydream of attempting trephination on myself.

<comment><USER5> But man do you guys make a killing. Holy astronomical salaries batman!

</comment></comment></comment><comment><USER5> I'm going to be really generic here and say:

1\. Learn a high-level OOP language. Java, C#, etc.

2\. Learn a functional language. Haskell, Lisp, ...

3\. Learn an imperative language. Such as C.

4\. Learn an "in between". e.g. SQL.

5\. Look for outliers. e.g. Aspect oriented programming, Eiffel (DBC).

Basically learn the different paradigms around programming and the common data types that span almost all languages. Learn when and where to apply those paradigms and the strength of each. Knowing those things will help you adjust to the future fairly well.

I'll give you an example of that in action. I learned some Common Lisp in my spare time. I then went on to use C# in 2000. So when Monads were eventually introduced in C#, I knew how and when to use them. My other colleagues at that time, their eyes glazed over for the most part. What was this new fangled contraption? It wasn't new at all, it was old school.

Knowing specific frameworks inside and out can be advantageous, but can also lead to a very narrow vision. If you make yourself well rounded, it leads to a very flexible skill set. You see something and you say "oh, that's just like X" and you dive in. I know you want to be a web development bad ass. But a lot of the knowledge to make you one is _outside_ of the web.

Then, if you're feeling frisky, dive into hardware.

<comment><USER6> Whilst learning these languages, don't just learn the syntax.

Get some good books on algorithms and data structures. Learn these really well and then study the languages to find out you can implement them.

Each language will lend itself to certain structures. Learn them.

<comment><USER5> You just reminded me of something I left out - patterns and anti-patterns. Knowing those might be just as valuable because then you can recognize and adapt quickly. For example MVC is basically the same across all platforms, with minor differences.

</comment></comment></comment><comment><USER7> Build something with no connection to the web whatsoever, eg. a platforming game that outputs ascii to the console, or a spellchecking engine, an emulator, or whatever. Make several versions of it, trying out radically different ways of conceptualising the problem in code. This stuff can sound intimidating if you've only done webdev, but it isn't at all as hard as it might seem.

Having a broader experience of the world of software will give you a wider range of skills and a wiser perspective on the web stuff that pays the bills these days. You'll be able to look at some horrible design and say "duh, that should be a finite state machine" and presto you've just wiped out a whole load of complexity.

Webdev is sort of a ghetto, the more "outside" knowledge you can bring to it, the more effective and the more employable you will ultimately be.

EDIT: also, become an automated testing deity ASAP. Just do it. You'll wonder how people can code without it.

<comment><USER8> how do you become an automated testing deity? Any recommended resources?

</comment></comment><comment><USER9> I think you'd most quickly benefit from [http://learnpythonthehardway.org/](http://learnpythonthehardway.org/) or [http://ruby.learncodethehardway.org/](http://ruby.learncodethehardway.org/)

It sounds like you haven't really moved much beyond googling for code snippets to solve specific problems. Work through one of the above books so you pick up some vocabulary and get introduced to a range of concepts.

A lot of the other comments are overly ambitious with their suggestions. Start small and learn Ruby or Python. I recommend staying away from frameworks such as Rails because of the high overhead of required knowledge that won't really benefit you as a programmer in general.

<comment><USER10> I completely agree, most people I've ever encountered asking "What to learn next" don't really understand what they think they know. There are 100 different directions you can go from if you actually had a decent understanding of php/html/css; exploring the PHP source code, learning more about the network protocols that http/https requests operate on top of, web application security and security in general, browser implementation of css rendering/javascript engines/sandboxing, all of these things are natural extensions of web development, and are really just the tip of the iceberg.

</comment></comment><comment><USER11> Personally, for me, it's been JavaScript. I started freelancing with PHP. The second to last freelance project I had was Java backend with AngularJS frontend, and the tricky bits were all in the frontend. Now I've been working freelance for the same company on several projects since June, doing 100% JavaScript; mostly frontend, with bits and pieces of nodejs, grunt and so on for tooling.

To get off to a good start I suggest learning how encapsulation and modules are done in JavaScript (e.g. prototype, the module pattern, possibly also node's exports or something like requirejs). This will allow you to actually build large JavaScript projects without them turning into a Frankenstein's "everything in a single file in jQuery's document.ready" monster.

Try to focus on learning the language and its features and not get lost in DOM-land (use libraries or your framework of choice for manipulating the DOM unless you have a very good reason not to).

Learn all about how to read and manipulate data using XmlHttpRequests, web services, REST, JSON and so on. This will be invaluable for any projects that have third party dependencies, which will be most of them these days.

Spend a little time learning about which frameworks are better for rich desktop-style browser apps (hint: probably the bigger ones, like Angular or Ember), and which are more suited to lightweight apps that will also run nicely in mobile or other embedded devices (hint: probably the smaller ones, like Backbone).

Polish your jQuery knowledge. It will be popular and useful for a long time yet, and it's a big library with lots of extremely handy functions and features.

Maybe pick up other useful "utility" libraries like underscore.

Finally, if you're interested in driving code quality and keeping clients happy and regressions at bay, I suggest you look into automated unit and integration testing. Again here there are large, complicated tools like karma and smaller, lightweight "test logic only" libs like jasmine. If you're like me, you will get lots of enjoyment and satisfaction figuring out how to wire up node, grunt, jshint, phantomjs and (your test lib of choice) to get a single command build/deploy/test script. If you're likely to work in a team, get that script running in a CI server for bonus points.

If you have any questions hit me up on email@example.com, happy to help! :)

<comment><USER12> I'm doing this right now - evaluating Angular vs Ember, then picking one to focus on learning well.

> probably the bigger ones, like Angular or Ember), and which are more suited > to lightweight apps that will also run nicely in mobile or other embedded > devices (hint: probably the smaller ones, like Backbone

Is it inadvisable to use Angular/Ember for mobile? I hadn't heard this before.

</comment></comment><comment><USER13> Honestly, get a job. What has taught me most about programming is having new problems to solve that required new skills, techniques and tools. The best way to "fund" your development is to find someone who will pay you to solve their problems.

I started out of school doin front-end, then the next project I started with frontend and then had to learn Django. Job after that was all Django/frontend. Then, Django to Rails. Now im doing Node/Go/hardware/etc. Each job paid me to learn new languages, adopt new techniques and find better ways to solve problems.

I'd hop on Craigslist or one of the millions of freelance sites and contact a few potential jobs that you think you can pull off but would require you to learn a bit. It may seem a bit haphazard compared to a more academic approach but it will be more of a realistic growth curve compared to the realities of the freelance world.

Also, side projects are a great way to experiment.

Think of it as "job driven development".

</comment><comment><USER14> If you have the financial resources to do it, I'd recommend spending some time exploring other parts of business and computer science. There are some really great free resources out there for learning (coursera, udacity, etc.). The reason I say that is that having some solid fundamentals will give you more flexibility in terms of what you can do, and it'll also let you try some other potential career paths on for size.

If you can spend a couple of years at the start building breadth in your skill base, you'll be able to gracefully adapt to changes in your lifestyle, the online market situation, and other things that come up. I know a couple of guys who jumped in full bore kind of like it sounds like you're talking about, and they eventually found themselves pigeonholed into a single career for their whole lives, because as life went on their obligations accumulated to the point where they could no longer afford the interruption in income that a career change would mean (because they would have to learn basically everything over again). So, if you can keep your expenses minimal and pay them with your freelancing work, I'd say just try to get some foundational business management and computer science knowledge under your belt.

As far as what tech/language/etc will give you the best bang for the buck in today's freelancing market I have no idea. Regardless of that though, I'd say that learning Internet marketing seems like the most important thing you can do to ensure success. This is something I failed to understand at the start and I'm paying for it now.

</comment><comment><USER15> Get serious about learning Rails, then get serious about learning EmberJS. If you are good at both of those you will be in a great place when it comes to building web apps.

</comment><comment><USER16> Looking for "the next big thing" is a crap shoot.

If what you want is all the consulting work you could ever possibly do, then learning more PHP will get you that. Stick with Wordpress and/or pick up Symfony, CodeIgniter, just about anything except Zend (and for the love of all things holy, do NOT go anywhere near Magento if you value your sanity).

If you're OK working a bit harder to find gigs but likely making more from them when you do (and facing less competition), Ruby/Rails or Python/Django would be good choices. You could go for more obscure languages and frameworks, but at that point you may have a hard time selling your services.

Starting with iOS (or Android) is also something to consider. People often don't realize that, when you get into full-stack web development, you have to be proficient with many technologies and software packages just to get a site working, tested, and deployed. While you'll likely end up having to expand into more than just ObjectiveC or Java fairly quickly when you work with mobile (don't get stuck in the PhoneGap/Appcelerator rut), a basic mobile app on either platform will be simpler to build and manage than a basic dynamic website.

</comment><comment><AUTHOR> Ok first of all, holy crap, I can't believe this actually hit the front page, definitely wasn't expecting that. Thanks again for all the input, I couldn't have asked for more.

Ultimately, I'm still not 100% sure, but after going through all the comments a couple of times, I think I've finally made up my mind. For the next month / couple of months, I'm going to slowly get back into freelancing while giving PHP a serious go. Although PHP may not be my first choice, I realise now this is actually the right one since WP development is going to be my main source of income for the next 6-12 months, so I need to get really good at it. Aside from that, I plan on sharpening my front-end skills and trying out a couple of other stuff as well such as SASS and Git.

After I feel I'm comfortable enough with PHP, I plan on trying out the Laravel framework, and see if I actually like it as much as Rails. Either way, I'm definitely going to learn Ruby as well ( I've already finished "Learn to Program" by Chris Pine some time ago ), and eventually spend less time on WP in order to finally master Rails. In the end, I'm hoping to give up WP entirely and work only on RoR projects ( either my own or by freelancing ).

I know a couple of you guys were suggesting that I should focus on things which are not related to web development at all, and I see your point, but honestly I simply can't get excited about that stuff at all. Maybe later on I'm going to feel different about it, but for now I'm focusing solely on web development.

Also, regarding SQL, that's something I may've left out in my original post, but fortunately I am pretty familiar with MySQL, I even studied it in school.

</comment><comment><USER17> I think you should first explore Computer Science in general. You should start with a good programming language, a high-level, beginner-friendly, very web- favored one. Pick Ruby or Python. Then grab courses on Algorithms, Databases, Computer Networks and an other area you wanna explore. You don't need to complete them, but just get a feel of what they have in store for you. Make sure you write some code as part of the Algorithms course.

Web development is not just using frameworks. There is real Computer Science involved for building and maintaining any substantially complex web-app. Apart from this, there will almost certainly be domain-specific knowledge required for a complex web-app.

If you get a hang of programming, and love the CS concepts you are trying to learn, you can then build a career path accordingly (may be college?). If you are dis-inclined, then you can get to learn Rails/Django, and keep free- lancing.

I think at the age you are in, you should not miss out on getting a chance to learn some serious CS.

</comment><comment><USER18> As someone in what feels like your exact same boat—though a bit older— I'd say just pick one. It makes the most sense to me, personally, to pick Python or Ruby or PHP (Laravel or similar) and learn how to actually program. I think choosing this path will introduce you all sorts of other stuff — rest, user input, security, modern front-end tools like LESS, etc.

Even sticking with WordPress could yield a good result for you. WordPress and PHP are constantly criticized for a number of different reasons, but the reality remains: WordPress and PHP occupy a huge chunk of a huge market.

I've personally identified this to be one of my biggest failings — not being able to actually program. At first I thought it was about language, and so I tried PHP (about 100 different times), I tried Ruby a few times, Python, and finally, JS. It turns out that all of these languages require the exact same thing — they require you to think like a programmer.

The furthest I got with it was with Ruby. I went through the entire Ruby course at Codecademy, read a lot of the Bastards Book of Ruby (fantastic), and even used Ruby to get through some Project Euler projects. By the end of a few good months of moderate input, I was absolutely still not a programmer, but I felt like I was beginning to think more like one.

So that's one piece of advice — if you want to learn to program, I think you actually need to decide on a language/framework and settle in and learn how to program. Learn how to think like a programmer, which is to say that it isn't about syntax and how each language does things slightly differently, but rather, is about process and patters and abstraction.

The other piece of advice is to be careful with freelancing. Paying the bills and learning to code don't necessarily go hand in hand. Spending 4 nights a week writing a WordPress theme for a client project really won't make you a better programmer. If I've learned anything, it's this. I've made some great supplemental money doing freelance work and I've learned a ton about WordPress, but I often think about the time I've put into it, and wonder what if I'd put that same time into learning and working on projects to further my ability to actually program.

And finally, the last piece of advice — try stuff out. Try JS, Python, Ruby, WordPress, Drupal, try setting up a VPS at Linode. Try everything you can to get sense of what feels right. I think that will help with your decision as well.

Good luck!

<comment><USER19> I have a couple friends that are trying to learn how to program and they are constantly bouncing from java to python to javascript to whatever else, and it really does them no good because they never sit long enough to learn the fundamentals of programming, just the basics of that given language. You really do have to work with one language for awhile and get comfortable with that language/environment and how to use it to solve problems. Once you get decent at this, then bouncing to the language du-jour is easier, more enjoyable, and you take away a lot more from it ('oh wow cool, in language X I had to do this, this, this and that, with language Y I just have to do this!')

<comment><USER18> Solid advice. It's taken me years of bouncing around just like that, but it's finally sinking in. =)

</comment></comment></comment><comment><USER20> It doesn't sound like you have mastered any one particular language. I would recommend pursuing one of the ones you have some experience in (Rails, PHP, or JavaScript) and making projects from scratch. Being able to build an entire application or utility from nothing and understanding all the parts is a great skill to have. I would not recommend learning any particular framework (web or otherwise) except as a tool to gain mastery of a language. Frameworks are scaffolding to allow you to use your understanding of a language to effectively build your idea into working models, they should not be treated as the be-all-end-all of a language. Limiting your learning to jQuery or Rails instead of trying to learn JavaScript or Ruby themselves will not get you the mastery you need to grow as a developer.

</comment><comment><USER21> Python, because it's easy to learn and it's super versatile. I don't code Python full time any more, but I still find a way to use it on a fairly regular basis. It's a very handy thing to have in your tool belt, regardless of what you wind up doing in the long term, and there's a lot of literature related to learning programming that's taught with Python. Python is probably the most average language you can find, which is why it works well for your stated purpose of having a more solid foundation. Although it's not the best tool for most jobs, it sounds like you don't entirely know what you want to do, and Python is an adequate tool for a _lot_ of different things; it can expose you to a variety of programming environments.

</comment><comment><USER22> I was in the same boat as you, and I swore I was going to learn Rails - <USER2> just 'go w/ it' -so I built an app in Rails..but when it came time to deploy -- my host only supported a specific version of Ruby -- and I wasn't about to pay $50 a month to host a hobby so to speak...

Then I decided I was going to try and duplicate it in php --which php is extremely ugly--UNLESS you're laravel. I found laravel which I am getting pretty darn good at, and I absolutely love it! It's Rails for php and has taught me GOOD design principles for php(instead of spaghetti code, like how to use namespacing and build my own packages, and their's a strong community for it -- I highly recommend going to #laravel on freenode(irc) -- for help when you need it.

</comment><comment><USER23> Think of a project, pick the tools, and start building. If you don't stop until it is done you will learn something.

I just did this with ruby and sinatra. When it was all done I ended up learning ruby, sinatra, javascript, jquery, redis, passenger, rack, haml, and how to edit gems in github.

Best of luck.

</comment><comment><USER24> I would study whatever you will study about programming plus study how to persuade people by using your writing or your speech. Study classic rhetoric (Greek) and modern persuasion techniques. This is very important to your success in business and in live. Learn how to make people do what you want them to do. This will help you when you are in a meeting with colleagues, when you are selling your software, even when you are designing your website. By learning how to persuade you will learn more about yourself and learn about the others.

</comment><comment><USER25> If you want to learn computer science more thoroughly, why don't you take a university-level course? You can continue freelancing to pay your way through it. That would give you broad-ranging skills and a good qualification to present to clients/employers.

I'm not sure if this is the best option for you, but then I'd be curious why that's not your first choice. Since you have a clear understanding of what you need, you can pick a course that fits you and ignore the bullshit that comes with it.

</comment><comment><USER26> The next big thing will be the next flop. I'd advise you to learn one of the good old things, like python.

</comment><comment><USER27> I too see JS growing to be an all around general purpose language, and it is definitely a good one to learn to break away from the standard class based languages. The book Javascript - The Good Parts is a great and explains the power and weaknesses of Javascript.

<comment><USER28> > I too see JS growing to be an all around general purpose language

Truly this is the darkest timeline.

<comment><USER27> Doomsday ;)

</comment></comment></comment><comment><USER29> I learned C at school and self-taught myself enough PHP and JS to do web development. I understand programming logic and OOP pretty well. Now I'm reading Code Complete by Steve McConnell to gain a better perspective on software engineering.

</comment><comment><USER30> Drupal. You already know wordpress, but with wordpress a fixed rate for a basic site runs $600, premium site $1800-$2000. There are premium+ sites that you get contract work for. But I've never heard of a boondoggle-scale Wordpress site. With Drupal that's all there is, boondoggle rates. I've never seen fixed rate sites done with Drupal, and there's a reason for that I suppose.

<comment><USER31> This cracked me up. But seriously, if you know some PHP and want to quickly be able to work on larger projects, Drupal is a great idea.

</comment></comment><comment><USER32> Whatever you do, pick one and OWN it.

</comment><comment><USER33> [http://www.codecademy.com/](http://www.codecademy.com/)

</comment><comment><USER34> If you went play framework w/ Scala it would be pretty hard to avoid learning some OO and you could also learn some functional stuff. You've got enough background to get over the initial hump, now start writing "interesting" stuff not just yet another CRUD app.

You seem to have a database sized hole... The hard part about databases isn't the syntax, or peculiarities of specific DBs, or even optimization tricks to make things faster, but design. What is normalization? Why/When would you want it? What tasks need a relational design, or not... Even if you never pivot into being a DBA it helps alot to at least minimally speak a DBAs language when you write a CRUD app talking to his DB.

Programming as in slinging code syntax stuff, or higher level design? Might want to crawl inside algorithms for awhile with Knuth and other books. Much like learning Algebra its not like you'll ever use it, its more to discipline the mind to figure out other complicated stuff. You should really google for and spend a lot of time at "project euler" if you're trying to learn higher level programming. Many of the PE problems aimed at turning you into a better mathematician can be hacked on brute force-ish to make you a better programmer. As a hint the first problem you're not "supposed to" brute force add those together, you're "supposed to" figure out the easy formula. But writing the brute force adder is none the less an interesting experience if you've never done it before in your language of choice. (edited to add, buy and read and "do" the entire "little schemer" series, or at least the first book)

I don't know if this would make you more employable, but in terms of extending your greater computer-ish knowledge you could do worse than some embedded stuff. Get an Arduino and some shields and some servos and some sensors and make it do something really well. You claimed to know a little bit of C so here's something fun to do with it.

You also seem to have an OS sized hole in your list of experience so some systems programming type experience might be interesting. Get a couple free machines (castoffs) and figure out how to use Puppet to make them jump thru hoops. Since you have a cluster, there's a lot of fun you can have learning how clustering/replication tech works and scales. Don't worry about using old junk computers, there's absolutely no difference between clustering on new big iron and on a free junkpile P3 other than the new stuff is faster. Make your own DB host and a bunch of front ends and see what happens. Maybe try a cluster of DB hosts and FEs talking to the DBs. Much like the DB thing you may never become a sysadmin but learning to speak their language will help even if you stick to webdev work.

</comment><comment><USER35> Where do you find clients?

</comment><comment><USER36> Using Eclipse to bootstrap into Java represents a pretty reasonable and comfortable environment to wade into.

Go to: [http://www.eclipse.org/downloads/](http://www.eclipse.org/downloads/)

Download the Java EE version. Create a project with a class named Main.java in a package like "com.example.test".

In the class:

      public static void main(String[] args) {
          if (args != null && args.length > 0) {
              for (int i = 0;i < args.length;i++) {
                  System.out.println("Hi there!");
              }
          } else System.out.println("Hello.");
      }

Hey, presto! You're a Java programmer. Try compiling and running it.

To get your feet wet with web programming, here's some quick and dirty bare bones code:

      TestServlet.java
      /******************************************************************************/
      package com.example.test;

      import java.io.IOException;

      import javax.servlet.ServletException;
      import javax.servlet.http.HttpServlet;
      import javax.servlet.http.HttpServletRequest;
      import javax.servlet.http.HttpServletResponse;

      public class TestServlet extends HttpServlet {

    	  private static final long serialVersionUID = 1L;

    	  @Override
    	  protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
    		  try {
    			  resp.getWriter().print("Hello. <form action=\"\" method=\"POST\"><button>Hey!</button></form>");
    		  } catch(Exception e) {
    			  System.err.println(e.getClass().getName());
    		  }
    	  }
    	  @Override
    	  protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
    		  try {
    			  resp.getWriter().print("Hi there!");
    		  } catch(Exception e) {
    			  System.err.println(e.getClass().getName());
    		  }
    	  }
      }
      /******************************************************************************/

And then...

      web.xml

      <?xml version="1.0" encoding="UTF-8"?>
      <web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd" id="WebApp_ID" version="3.0">
        <servlet>
          <servlet-name>main</servlet-name>
          <servlet-class>com.example.test.TestServlet</servlet-class>
        </servlet>
        <servlet-mapping>
          <servlet-name>main</servlet-name>
          <url-pattern>/</url-pattern>
        </servlet-mapping>
      </web-app>

This will compile and run on Apache Tomcat, if you compile and deploy as a ROOT.war file.</comment>


A collaborative spreadsheet in less than 45 lines of JS, one library used - kav-ya http://jsfiddle.net/sy85U/ <comment><AUTHOR> It would be nice if people could lay off the script injections. It's clearly insecure, but that's not really the point. It makes the experience worse for everyone if you alert(), etc.

<comment><USER2> > It's clearly insecure, but that's not really the point.

No, that is the point. I even pointed it out hours ago: [https://news.ycombinator.com/item?id=6727448](https://news.ycombinator.com/item?id=6727448)

The failure is the lesson. The previous version was a clever hack written by a clever person. This is ignorance, and the lesson is that allowing users to run arbitrary code on other user's computers is bad idea.

<comment><USER3> I'm pretty sure that OP knew the issue, not just you, so it's not ignorance. And it's not the moment to cite The Good Parts either.

There is no failure here to be seen. There is a clever hack to make a spreadsheet shared quick and dirty.

Just a reminder that all the JS code you run, in particular on jsFiddle, is untrusted and is part of the security model of the JS engine in your browser that evil JS code must not be able to do any harm. If it did, report it to the browser vendor and earn a bounty.

</comment></comment></comment><comment><USER4> If the spreadsheet doesn't load you can watch the chaos unfold in real time by visiting the datastore's page directly: [https://spreadsheet.firebaseio- demo.com/sy85U](https://spreadsheet.firebaseio-demo.com/sy85U)

</comment><comment><USER5> Nice. I look forward to the full office suite in 60 lines of JS, the email client/server in 75 lines of JS, and of course the bitcoin exchange web app in 90 lines of JS.

<comment><USER6> Did you forget an operating system in 150 lines of JS? (Four libraries used)

<comment><USER7> Not sure how many lines of js it uses, but here you go:

[http://bellard.org/jslinux/](http://bellard.org/jslinux/)

[http://bellard.org/jslinux/tech.html](http://bellard.org/jslinux/tech.html)

</comment></comment></comment><comment><USER8> I am waiting for someone to reproduce healthcare.gov with 100 lines of code...

<comment><USER9> that's the spirit ! seriously it would be an awesome idea to create a website with challenges like that. create a substractive synthesizer in js in less than 1k, like old demos

<comment><USER10> [http://js1k.com/](http://js1k.com/)

</comment></comment></comment><comment><USER11> I think the title should warn users of the various remote injection vulnerabilities present in the script. It took about 5 seconds for the page to change to xhamster

<comment><USER12> Which would have been extremely embarrassing for someone at work. Luckily, that didn't happen to me.

</comment></comment><comment><USER13> Wow this link took about 30 seconds to turn into porn. Be warned.

</comment><comment><USER14> WARNING: Could there be some kind of script attack ? My Safari browser freezes with 'foo' alert message from this site and it has placed itself as default website so whenever I reopen safari it freezes again.

Let me know if you know how to fix this.

<comment><USER15> I guess you could start Safari while offline to prevent the page from loading and change back your default page to something sane?

</comment></comment><comment><USER16> Collaborative eval with the world, I'm glad I wasn't logged into jsfiddle when I opened that.

<comment><USER3> Since jsFiddles run user-generated code by design, I hope that they correctly sandbox and use HTTP-only cookies anyway.

<comment><USER16> That's a great point. A restricted iframe on a separate domain + httpOnly session cookies just to be safe would do it.

<comment><USER17> That's what they do already.

</comment></comment></comment></comment><comment><USER18> Now I really love the feature of Chrome, "Prevent this window from creating new dialog box" ( or something like that ). Some *$%#@ put alert in the cell.

</comment><comment><USER19> Careful, this code runs eval() on all spreadsheet fields. Someone can "collaboratively" steal your jsfiddle.net cookies.

<comment><USER17> The code runs on the fiddle.jshell.net domain.

<comment><USER19> that's good :) did not realize

</comment></comment></comment><comment><USER20> I like the craziness that this devolved into. It's funny when a bunch of people are all editing like mad.

I had the thought that it'd be fun to have a contest using jsfiddle to start from some point, like the excel (lite) clone in 30 lines, and add the best/coolest feature in some limit of lines.

It's really wonderful how ingenuity stacks.

<comment><USER21> Such hax.

I wonder if a collaborative drawing app could be made with this, using canvas.. I keep trying to figure out exactly how it works but then sparkleponies and alerts everywhere...

<comment><USER22> Not Firebase, but you might enjoy: [https://hacks.mozilla.org/2013/10/introducing- togetherjs/](https://hacks.mozilla.org/2013/10/introducing-togetherjs/)

</comment></comment></comment><comment><USER23> I think I crashed it (ie: Chrome hard lockup on the tab) with this:

=location.href='google.com'

<comment><USER2> That will just throw a frame error because of the X-Frame-Options header, which is caught quickly.

Browsers tend to have more problem with =while(1){}

</comment></comment><comment><USER24> Somebody has created a bot to put script lyrics in the spreadsheet ._.

<comment><USER25> Is it a bot or injection that maintains itself?

</comment></comment><comment><USER26> Sweet. How do you do I make it so not every Joe can hack the sheet?

<comment><USER27> You'd have to parse the cells as some sort of DSL that only allows mathematical tokens, as opposed to eval which allows access to the full arsenal of the JS language. But I think excel is Turing complete

<comment><USER28> Brings up the interesting question -- I wonder what turing-complete DSLs have drop-in JS libraries you could use to replace the call to "eval" here.

<comment><USER29> [http://nayuki.eigenstate.org/res/brainfuck-interpreter- javas...](http://nayuki.eigenstate.org/res/brainfuck-interpreter- javascript.js)

</comment></comment></comment></comment><comment><USER30> Oh boy. I can't wait to see what someone else can do with 60 lines of JS, and two libraries used!!

[Note: not being sarcastic. Think this is a genuinely awesome way to respond to first post!]

</comment><comment><USER3> I would love to read about who/how is blocking XSS and censoring!

<comment><USER17> Oh, that's me: [http://jsfiddle.net/sy85U/31/](http://jsfiddle.net/sy85U/31/)

Just a very quick, crude little hack.

<comment><USER3> You know what would be cool? Injecting this modified version to all clients, so that everybody act as a censor.

</comment></comment></comment><comment><USER31> [http://images.retecool.com/uploads/reet- And_its_gone_origina...](http://images.retecool.com/uploads/reet- And_its_gone_original.jpg)

</comment><comment><USER32> I think we broke it, the fiddle no longer works for me

<comment> kav-ya Try clearing your local storage: [http://stackoverflow.com/questions/7667958/clear- localstorag...](http://stackoverflow.com/questions/7667958/clear-localstorage)

</comment></comment><comment><USER33> I like that everyone is collaborating to keep N S A in the last columns, that is team work !

</comment><comment><USER34> very cool, probably need a few 100 more lines of code to stop XSS but awesome none the less.

</comment><comment><USER35> I'm trying to remove the location.href thing outta there! Annoying!

</comment><comment> cm-t Ooups, someone has having success with XSS :/

</comment><comment><USER36> lol, I think I just got hacked from that

</comment><comment><USER37> Why not just use TogetherJS securely?

</comment><comment><USER38> did someone just execute remote javascript`?

</comment><comment><USER39> someone just broke everything hahaha

</comment><comment><USER40> And someone killed it...

EDIT: Nevermind, it's back.

</comment><comment><USER41> Dope.

</comment><comment><USER42> wow it's alive</comment>


The Electoral College Is the Greatest Threat to Our Democracy - <AUTHOR> http://www.nytimes.com/2019/02/28/opinion/the-electoral-college.html <comment><USER2> The democratic republics that score the highest/best in things like quality- of-life, income equality, incarceration rate, crime rate, access to healthcare, life expectancy, etc., _all_ have forms of government where the executive is _not_ elected at all by popular vote, but is instead chosen by the legislature.

The only other countries that have a form of government like the US's (a presidential republic) are ones like El Salvador and Turkey.

<comment><USER3> Yes, I always find it funny when I see just how easy it is to pass a vote of no confidence in parliamentary systems and have a new executive government set up. While in the US you have to wait four years(!) if you want a new government even if the majority of people do not want the current executive government in power.

<comment><USER4> Even in a parliamentary system the majority of people may not want the current government in power if the parliament they elected years before does.

<comment><USER2> It's still much better than the presidential system, because you don't get deadlocks between the executive and legislative branches that last indefinite amounts of time. The legislature picks the executive (instead of the voters electing him separately and on a different cycle), so the two branches tend to be in agreement. And the voters are able to change the makeup of parliament to some extent at every election.

Also, parliamentary countries seem to have elections based much more on party rather than actual people, so they have more parties with representation, instead of devolving to just two parties per Duverger's Law.

<comment><USER4> > It's still much better than the presidential system, because you don't get > deadlocks between the executive and legislative branches that last > indefinite amounts of time.

Sure, but that's a different argument.

> (instead of the voters electing him separately and on a different cycle)

Nothing prevents having both elections on the same cycle. AFAIK France (which is semi-presidential) somewhat recently aligned their cycles to make that disagreements between president and parliament less likely.

> Also, parliamentary countries seem to have elections based much more on > party rather than actual people, so they have more parties with > representation, instead of devolving to just two parties per Duverger's Law.

That actually weakens the claim that parliamentary systems are superior, unless presidential systems somehow cause first-past-the-post.

I think they may have a common cause though: If the writers of a country's constitution believe that people should vote for parties, they'll probably chose proportional representation and a parliamentary system. If they want people to vote for individuals, a presidential system with FPTP is the obvious choice.

</comment></comment></comment></comment></comment><comment><USER5> Are we not a Republic? And the Electoral College was put specifically into place so as to avoid devolving us into a Democracy?

<comment><USER4> The Electoral College was put into place to avoid the USA devolving from a non-monarchy to a country ruled by its people?

<comment><USER6> > The Electoral College was put into place to avoid the USA devolving from a > non-monarchy to a country ruled by its people?

The US at the time was a slaveowning nation for which voting rights could only be held by white, male, landowning gentry... so yes.

</comment></comment></comment><comment><USER7> A counterargument (with a mathematical basis in a paper I could not find) is described at [http://discovermagazine.com/2004/sep/math-against- tyranny](http://discovermagazine.com/2004/sep/math-against-tyranny) . The NYT also it in [https://www.nytimes.com/1996/11/03/weekinreview/why-the- elec...](https://www.nytimes.com/1996/11/03/weekinreview/why-the-election-is- like-baseball.html)

</comment><comment><USER8> It's interesting that each State can choose its electors in whatever way they want and have all (with a few exceptions) converged on the winner-take-all popular vote of the State's voters. I think if a State gives the electors to the national popular vote winner but that State's voters chose another candidate, that policy won't last long in that State.

<comment><USER9> It seems to me that any effort to subvert the Electoral College will be overturned on constitutional grounds.

If folks want the Electoral College abolished, get an amendment passed... ;-)

</comment></comment><comment><USER10> So is a massively biased press establishment.

<comment><USER11> Yeah, but it is an opinion column after all, and labeled as such.

Some (many?) "news" outlets have opinion makers speaking off the cuff all the time and don't bother to label it as such.</comment></comment>


Show HN: PhotoDog – Easily compress images at optimal quality in seconds - <AUTHOR> https://photodog.bakchoi.com/?ref=hn <comment><USER2> What's privacy like on this thing?</comment>


You can use dots (periods) anywhere in your Gmail username - <AUTHOR> http://samengland.com/you-can-use-dots-anywhere-in-your-gmail-username-as-many-as-you-want/

<comment><USER2> If your email address is N characters, there are actually 2^N possible email addresses by just adding dots in different places (due to the binomial series).</comment>


Rep. Heck: Why Should Americans Care About Russia Meddling in Our Democracy? - <AUTHOR> https://www.youtube.com/watch?v=k7Hh5aXybVY <comment><USER2> Is the fact that this person is a representative proof that Russia has been meddling in our democracy for a long time? Or have we, all by ourselves, managed to elect as representatives some people who are this badly in need of a civics lesson?

I'm not sure which hypothesis I find more disturbing.

<comment><AUTHOR> I feel like you didn't watch the video.

<comment><USER2> I'll be honest - I didn't. I'm at work, and they want us to not load down the network with video during work hours.

I gather that I was misled by the title...</comment></comment></comment>


Show HN: Make your keyboard sound awesome again (Model M Sound Emulation) - <AUTHOR> http://github.com/adewes/awesome-keyboard Make sure to check out the Youtube video as well:
http://youtu.be/-eANf3QWStU
Enjoy :) <comment><USER2> possible for mac?

<comment><AUTHOR> in principle yes, right now I use xlib to capture key presses, which is probably not available there, but it should be easy to replace that.</comment></comment>


Show HN: FOS – Web components to turn your web app into a fake OS - <AUTHOR> https://github.com/victorqribeiro/fos <comment><AUTHOR> This project started with me studying web components. After a short while I decided that I wanted to do a "fake operating system", so I merge them both.

I also using digital ocean now, so people trying to access the live version should have no problems anymore. The icons used in the project are free to use with proper credits, so I did not include them on GitHub, but the code I used to create the live example is pretty much the same on the tutorial I wrote.</comment>


Windows 3.1 All Over Again - <AUTHOR> https://tomjoro.github.io/2017-02-03-why-reactive-fp-sucks/ <comment><USER2> The real "Windows 3.1 all over again" experience is mixing functions which block with ones that are "async". If nothing blocks for long, as with Javascript, that's fine. If everything is threaded, and threads block, that's fine too. What works badly is mixing the two. If you block something that's doing cooperative multitasking, the whole system stalls. This is still a big problem inside browsers.

Python has this problem with "async". If you make a blocking call, you stall out the "async" stuff. Go gets around this by having a "fiber" or "green thread" mechanism underneath. If you block on a lock, that doesn't tie up a resource needed to run other goroutines. This is why retrofitting async to a language has problems.

<comment><USER3> Note that Python has greenlet and gevent, though. It works by monkeypatching all blocking calls in the standard library (and common external libraries such as database client libs like psycopg2).

<comment><USER2> That's not a standard part of the new "async" feature. It's a leftover from the Stackless spinoff of Python.

</comment></comment></comment><comment><USER4> This feels like Elixir propaganda =/

Starts off nicely with a good premise, but instead of digging into the interesting details it glosses over them and starts to preach Elixir.

I don't have any stake in this, I use neither js or Elixir, but this article is too shallow to be really useful.

<comment><USER5> Yeah he stakes a bit of a stab at scala play, which as far as I can tell in the latest release uses the same concurrency mode (actors) as Erlang anyway

</comment></comment><comment><USER6> I tend to think of it as a pre-emption issue more than a concurrency or parralellism issue. Endless loops cause hangs. Being able to pre-empt things would solve a lot of the problem.

        10 print "mike is cool"
        20 goto 10

simply doesn't have a good analogue in javascript. Even in Web Workers a loop like that causes the worker to be unresponsive to events. The Shared Memory addition alows a rudimentary form of this by allowing a flag to make an aware program exit. It wouldn't be too hard to have a BASIC to worker compiler that compiled the above code to

        while (true) {
           print("mike is cool");
           if (shared_global_flag === true) break;
        }

There are a few javascript livecoding websites out there that update the code as you type. There is a problem with this. Broken (as half written code tends to be) javascript doesn't just fail to work, it breaks other things

for example

        var x=1;
        while (x<10) {
          output(x);
          x+=1;
        }

Typing this into a livecoding environment breaks because the while loop goes on forever until you type the code to increment x. It breaks everything because the while loop blocks you from being able to type the code that would get you out of the situation.

I don't see the wisdom of disaallowing something due to the specter of race conditions when its absence seems to cause a much easier to cause debilitating problem.

<comment><USER7> apparently codepen runs the code through a parser and inserts exit checks in all detectsble loops.

[https://blog.codepen.io/2016/06/08/can-adjust-infinite- loop-...](https://blog.codepen.io/2016/06/08/can-adjust-infinite-loop- protection-timing/)

<comment><USER7> as for why no races I think because it's basically impossible (or way too non performant) to write a JavaScript engine that can handle multiple threads deleting properties on objects. You'd have to lock on every delete.

<comment><USER6> That's the nature of JIT though, non-performant but working is ok for the bits that might cause trouble. JavaScript should just make fast the things it can.

I'd be ok for a light pre-emtion that would happen between javascript statements. Instead of genuain preemtion have the Javascript engine include break checks. A check of a flag and conditional branch (usually not taken) is not a huge expence. The cost can be especially ameliorated if you do the check only on statements that jump to earlier statements.

That fixes the problem of long running code stopping other things. It doen't need system level threads or parallelism, just a js engine doing spot checks.

</comment></comment></comment></comment><comment><USER8> Author seems to slightly miss the point of Node. It's designed for IO-bound work, and essentially nothing else.

If you have a long-running synchronous algorithm, you should not be running it in Node, or alternatively you could dispatch to another process/C lib and have it run in a true thread and asynchronously wait for the result.

<comment><USER9> > Author seems to slightly miss the point of Node. It's designed for IO-bound > work, and essentially nothing else.

No remotely relevant app is only ever IO-bound. There are always some long lists to sort, some graphs to walk, some intricate canvas renderings to do.

<comment><USER10> I'm not sure about that assertion.

I work on a giant [Health|Fin|Defense]Tech monolith which has been around forever, has do to everything for everyone, and has been worked on by hundreds or thousands of developers with radically different skill levels. It connects to many databases, external services, etc., and does some immensely complex data munging just to render what you'd think are simple pages (since the inflexibility of the backing model and limited space to make denormalizations of really big data mean everyone has to do super complex aggregations in the app server, across data sources X, Y, and Z, all to show the user a 10 row table).

In short, it's huge, ugly, and computationally expensive.

I was asked to quickly research the benefits of switching its platform (a single-threaded scripting language) to NodeJS (I wasn't told to research anything other than NodeJS, despite my objections).

I figured the savings would be minimal, since all our application servers are constantly running out of CPU (page loads crazy expensive, see above) or mem (aggregations crazy expensive, see above).

So I broke down what the app was doing on some representative servers, working both from the coarse level (dtrace/system resource usage) to the fine level (flame graphs of calls/wait time/yield events within the application runtime itself). I didn't profile the batch processing services; they were RPC'd to via the renderers, and used more appropriate languages/patterns for huge-data manipulations. As far as my profiling was concerned, they were functionally databases.

The result? 88% on average time spent waiting on IO or blocking non-block-file system calls. P90 was 99% blocking.

That went totally against my assumptions.

Sure, our webservers were overloaded with non-IO load, but if we were to switch to non-blocking IO and buy more webservers, we'd have gotten a massive performance increase _without_ having to change the fundamental architecture of our webapp.

That was when I started seriously considering the benefits of reactive-style programming in a single thread, a la NodeJS; it hits a nice balance between "programmers that aren't necessarily super skilled having to engage with a full/real concurrency system" and "do everything blocking one per process".

There are tons of downsides, of course. Switching to nonblocking IO after spending so long in a blocking world would require both massive technical expenditure, and would probably also require reorganizing the capacity planning of all the other services/databases the app servers talked to, since they'd be fielding a lot more requests. Basically, the blocking nature of the render loops was an informal rate limiter on database queries. Parallelizing the render loops via processes gave much more direct control of changes in resource utilization, which is nice for proactive scaling. Additionally, node/callback style is still harder to learn (even with async/await sugar) than plain 'ol top-to-bottom sequential code. All that said, we'd be rewriting code in a new platform that _looked_ different, but the code could still _do the same things, per render, in the same order_ , which is a huge benefit.

A platform that hides preemption/concurrency while allowing people to program in the sequential style (e.g. Erlang) _might_ have been a better fit, but . . . we were already using one of the best M:N resource schedulers in the world, the Linux process scheduler, to multiplex concurrent sequential processes that were just . . . linux processes. At the end of the day, I gained a lot of respect for the power and balance struck by single-thread/event-loop-driven reactive runtimes like Node.

Edits: grammr.

<comment> le-mark _So I broke down what the app was doing on some representative servers, working both from the coarse level (dtrace /system resource usage) to the fine level (flame graphs of calls/wait time/yield events within the application runtime itself). I didn't profile the batch processing services; they were RPC'd to via the renderers, and used more appropriate languages/patterns for huge-data manipulations._

That's very interesting; you were given a huge old legacy app that had scaling issues (cpu and memory). Presumably the business was tired of throwing hardware at it? Or they hadn't gotten to that stage yet? Continuing, and tasked with diagnosing the performance problems you looked at the system and application. I can read up on dtrace, but how did you profile the application level stuff (time/yield)? Was it some functionality provided by the run time? Ie java visual vm for example?

This is a problem many, many companies have; ill performing legacy apps that the "legacy" staff aren't capable of handling (because the talented people left long ago ie "don't move my cheese"). It'd be really educational to see a write up of this!

<comment><USER10> It broke down roughly like this. I can't write it up, and am being vague, because I don't wanna get yelled at, sorry. Googling the below techniques will get you started, though.

1\. Simple resource usage (system time vs. user time, memory, etc.) got the metrics for how long the OS thought the app was spending waiting on IO.

2\. Dtrace was able to slice those up by where/how they were being called, which syscalls were being made, and what was being passed as arguments. This was important for filtering out syscalls that would remain a constant, high cost (e.g. block local file operations on old versions of Linux, which we have, get farmed out to a thread pool in NodeJS, so I pessimistically budgeted as if that thread pool were constantly exhausted due to volume + filesystem overuse).

3\. In-runtime profiling. We have the equivalent of java visual VM (well, more primitive; more like jstack plus some additional in-house nice features we built, like speculative replay), but for our scripting language platform. That generated flame graphs and call charts for processes. Those were somewhat useful, but tended to fall into black boxes where things called into native code libraries, which was where the dtrace-based filtering data was able to help disambiguate. Using this we got a comprehensive map of "time spent _actually waiting_ for IO".

There was a lot more to it than that, though:

Since different syscalls both had different call overhead (and different call overhead depending on arguments supplied) and different blocking times, all 3 steps were necessary.

For example, an old monolithic chunk of code that did ten sequential queries to already-open database connections is going to issue select(2) (or epoll or whatever) at least ten times. Conversion to Node, and it's single-poll-per- tick model, would make that cost vastly reduced, making it move the performance needle a lot. Of course, that's only true if the ten queries in question can _actually_ be parallelized, which typically requires understanding the code . . . if it can be understood, which is not a given.

However, a page render that called ten different HTTP services would make ten full-cost connect(2) calls in the worst case, ten low-cost (keepalive'd) connect calls in the best case. Node would still have to make those same ten calls, making it a less needle-moving thing to move into nonblocking IO (though the time spent waiting for connect to complete or time out would still not be paid directly in the render, which had to be accounted for as a positive). And it goes deeper: depending on the services being hit, keepalive window, and rate at which they were called during a typical server's render operations, we had to calculate how often, say, a 50-process appserver worker pool would be _redundantly_ connecting to those services (because separate sibling processes can't share the sockets if they're initiating the connection, and before you ask, I would not like to add to the chaos by passing open file descriptors over unix sockets between uncoordinated processes thank you very much). If the redundant connect rate was high, Node might offer significant savings by allowing keepalive'd sharing of connections within a single node process (we'd need many fewer of those per server than appserver workers). If it was low, fewer savings.

TL;DR it's complicated but possible to measure this data using established techniques. You don't have to get super far down the rabbit hole to get a _decent_ guess as to whether it will be beneficial to performance, but transforming _decent_ into _good_ requires a fairly thorough tour of Linux internals.

And, as always, the decision of whether or not to switch hinged primarily on the engineering time required, not the benefits of switching. C'est la vie.

</comment></comment></comment></comment></comment><comment><USER11> The author seem to loose the point of FP, as is the case with many web programmers

Async != FP

Back in my high school days, FP was almost strictly about usage of pure functions above anything else.

The late comers to the party, the React guys, then bunched FP with async stuff, and function polymorphism. That was wholly apocryphal to the original FP idea.

I can still cite my high school teacher citing Niklaus Wirth: "If a function does not interact with external data in a any way other than reading them from its inputs, the whole program can be represented as a spreadsheet"

The original advantages were states as:

1\. a program written in this manner is very easy to optimize. 2\. an interpreter or a compiler executing such program is easier to program, and is less error prone 3\. for as long as a kind of middleware sits in between the "spreadsheet function," and the data, it is possible to write a continuous execution log with ease, as well as to do tricks with getters/setters to start execution from arbitrary points in code.

All of that is stuff from times when Oberon and Modula were "hip" (seventies)

<comment><USER12> Oberon was designed in 1986 and it was hip up to around 1998. :)

</comment></comment><comment><USER13> > "what else am I supposed to do when I'm waiting for a result from the > database"? I can't do anything else until I have that result. I need the > result to make give to a template to give to the renderer, etc.

I dunno, maybe you could serve one of your 100 other concurrent requests?

> You can do something useful in the meanwhile and get the result later

and then immediately...

> It has nothing to do with hogging or stealing resources.

It has everything to do with this. If you're blocking on something waiting for a result while you could do something useful in the meantime, that is the definition of hogging resources.

<comment><USER14> His point is that blocking while waiting should not be an expensive thing. In preemptive systems like BEAM, it's not. Blocking a process is not an issue. In Node/Windows 3.1, it is a big deal, as it means starving other work.

<comment><USER15> Await works the same way as BEAM, no?

In theory their performance and behavior should be the same. BEAM and other N:M green thread systems being maybe faster if they can efficiently use multiple threads. But since they use message passing anyway (needed for isolation), they are again almost equivalent to running multiple Node/reactive request handlers.

N:M is probably useful for desktop apps. But for that single thread is (should be) always enough, so same behavior.

<comment><USER16> "In theory their performance and behavior should be the same."

This isn't about performance, it's about how you write code. In Erlang/Elixir (and also Go and Haskell), you aren't sitting there constantly explaining to the runtime over and over again how to plumb together bits of "async" code. Essentially, all code is simply presumed async, and the plumbing is always already there for you; the act of calling a function includes, for free, all the "async" plumbing.

I expect anyone who has been doing this for a couple of months should have noticed it's the _same_ plumbing, over and over again. Here's the long-running IO task. Here's what to do with it when it's done. Here's what to do with errors. Here's the next long-running IO task. Here's what to do with it when it's done. Here's what to do with errors, which is pretty similar to the last thing. Here's the next long-running IO task. Here's what to do with it when it's done. Here's what to do with errors; whoops, forget that one, which is gonna cost me during debugging but nothing in the code, runtime, or syntax affordances particularly cares that I forgot. Here's the next long-running IO task....

Yes, they do come out roughly the same in speed for Erlang/Elixir and JS, and Go can quite easily be substantially _faster_... _and_ easier to write.

And if we want to discuss performance, Go still isn't as fast as it could be. There's a solid reason why N:M threading can't be the fastest way to run code in general, but I don't see much reason why it couldn't be something more like 20% away from C speed in general, with some pathological cases where it's much slower and out of the question (very fast packet handling with those fancy software TCP cards, for instance).

<comment><USER17> async/await as mentioned by the previous poster is "do-notation" for the asynchronous monad Promise<T> in modern ES/JS(/TS). It's not terribly different from the languages you mentioned, even if it is a different approach because the language started synchronous first. You don't have to do that much plumbing as with the bad old JS world of Node callbacks, just sprinkle the async/await as appropriate, make sure things stay in Promise<T> and write things do-notation style just about like you would write the equivalent synchronous code.

From what I've done async/await in ES (and C#) is easier to write than Go's equivalent, but we're hitting your mileage may vary territory.

</comment></comment></comment></comment></comment><comment><USER18> Can we stop with the erlang/Elixir is slow? For most web services it is as fast as go.

For my last side project I was evaluating go and erlang. Just to test out how faster go is, I quickly coded a simple HTTP request in both language(I am not an expert in either).

The server would get a request with large body, cache it to a file on the disk, then load it in 128k chunks, get the hash of the chunk and save both the digest and the 128k data to a database.

To my surprise, erlang was consistently slightly faster for each request. Happy to share the code snips of both if anyone want to audit/scrutinize.

Edit: Code for both: [https://pastebin.com/aBQWqkG3](https://pastebin.com/aBQWqkG3)

On my laptop a single request(1.2MB) takes around 230ms for Erlang and 280ms for Go. Not a scientific test, but it gives an idea.

<comment><USER19> Your Go code uses "defer" in a for loop, which could be a performance bottleneck. The deferred function call doesn't run until the saveToDb function returns. A simple fix would be to reuse the rows variable and put the defer outside the loop.

I believe it might, depending on the database driver, also cause another issue. Since you're not closing the rows right away, and not even consuming the rows, this may force the driver to mark the internal connection as busy until the defer runs, meaning the next database call would have to open a new connection. Try closing the rows as soon as you can.

If you don't expect any results, you can also use Exec instead. Then there's nothing to close.

<comment><USER18> Just to update that you where right. Without the `defer` now golang is as fast as erlang.

Exec vs Query didn't make a difference.

Yet another update: things get really interesting when I run _wrk_. I don't want to spoil it for you. I can let you try it ;)

<comment><USER20> Please spoil it for us. I would love to see results, but can't run wrk myself at the moment.

<comment><USER18> I get 2x more requests on erlang

<comment><USER18> OK, I was unfair to golang again. Turns out it needs a bit more tuning "db.SetMaxOpenConns(10)" and now golang and erlang are on par.

</comment></comment></comment></comment></comment></comment><comment> le-mark This articles is a fantastic, real world overview of cooperative and preemptive multitasking. Note that coroutines and fibers are essentially equivalent because they can be implemented, each with the other[1]. Unfortunately this thread (up to now) has been simply language tribal-ism. Helps if everyone is on the same page with respect to technical terms before that all starts.

[1] [https://stackoverflow.com/a/3325985/2561675](https://stackoverflow.com/a/3325985/2561675)

</comment><comment><USER21> There are other concepts, like coroutines. At work, I use Lua coroutines to handle network requests. Basically, any function that can block instead starts the IO request, then yields the current coroutine. Yeah, the management code was a bit of a pain to write, but you can write (using the BASIC example from the article and not Lua, but the idea is similar):

        10 A = 1
        20 A = A + 1
        30 PRINT "HELLO " + A
        40 PRINT "YES"
        50 GOTO 20

where `PRINT` will do a non-blocking write, if there's anything left over, set the IO descriptor to trigger when ready for writing, and yield(). It's in the poll loop (select(), poll(), epoll(), kevent(), whatever the API is) to schedule the proper coroutine.

Yes, this requires making sure the code doesn't do anything CPU intensive, but for the application I have running, that is the case. But for me, using Lua coroutines means the main business logic is still sequential, even in an event driving environment.

Also, because you call a function to preempt execution, you don't need to save the entire CPU state. For C on the x86-64 bit, this is just six registers: [https://github.com/spc476/C-Coroutines/blob/314607cf058352be...](https://github.com/spc476/C-Coroutines/blob/314607cf058352bee922a4c0d1cfc0a37646d250/coroutine_yield-x86-64.asm#L85)

</comment><comment><USER22> >In Elixir (thanks to Erlang) yo have true pre-emptive multi-tasking

>An Erlang process is not a system process. An Erlang process is a lightweight process. But here's a key difference: there is no way in Erlang for processes to share memory

I'm confused. Is there real concurrency in Erlang? If yes, what Erlang processes use under the hood: system processes or threads? If threads, why they aren't sharing memory?

<comment><USER23> The VM has a set of threads called schedulers, all running in parallel. Each scheduler has in turn a set of Erlang processes (possibly thousands) to run. Every process has a pointer to its own separate chunk of heap where it allocates its stuff.

The language has no concept of pointer, so there is no way to create a reference to memory owned by another process. You can send a message to another process, but internally the VM will copy the message to that process' heap.

</comment></comment><comment><USER24> "I've heard a few reasons:" ... that you've considered, then cherry picked two of them, so you can confirm your views?

To properly reason why the approach is wrong, shouldn't you consider all significant reasons, including backwards compatibility as probably the biggest one?

"New language that learns from the mistakes of languages before it" would generally be better than "language and runtime that is keeping compatibility with a programming language that was heavily rushed just to fill a feature point for a web browser"

One might as well state that the grass is green and sky is blue, no?

</comment><comment><USER25> With all due respect, "the reasons he heard" are bogus.

The only reasons there is no preemptive multithreading in Node is just that the V8 interpreter is under GIL.

</comment><comment><USER26> " 10 A = 1 20 A = A + 1 30 PRINT_NON_BLOCKING "HELLO" \+ "A", when done callback { LINE 40 } 40 PRINT "YES" 50 GOTO 20 A common quiz for Javascript is to ask: which is printed first? "YES" or "HELLO3"? "

Is that a mistake with "HELLO3"? Should it not be HELLO2? Or am I stupid?

<comment><USER3> Please note that your formatting is totally screwed, making your fine comment almost unreadable.

See also the "help" link next to the edit box: [https://news.ycombinator.com/formatdoc](https://news.ycombinator.com/formatdoc)

</comment></comment><comment><USER27> With nodeJS using fs.readFile is basically the same as scaling across multiple machines. NodeJS kinda force you to learn how to manage concurrency, eg what people call callback/promise/future hell.

</comment><comment><USER28> > Concurrency is hard. If you want concurrency you have two choices - > processes or threads, take your pick.

Sometimes there's a 3rd choice: actors. imo it's much easier to manage than threads

<comment><USER19> In Erlang, actors are called "processes".

</comment></comment><comment><USER29> yup... I think browser need to come with some type of bytecode or vm that supports concurrency so we can finally have a true app platform.

<comment><USER13> Don't you already have this with WebWorkers? This isn't a limitation of the environment per se as much as it is a limitation of the DOM.

<comment><USER30> The DOM wants to be accessed sequentially by a single thread, then we ended up to say single threading is good because we don't have alternatives and because we're using the same frontend technology for backend jobs (v8.) It's a kind of Stockholm syndrome.

Sequential access to the DOM can be ok because we are the only user of the browser. Single processing is not so ok on the backend because there could be thousands of users there. We scale Ruby, Python and Node with multiple processes (I'm doing it.) I'm also developing an Elixir application using Phoenix. The approach is similar to writing Rails or Django code (I never sent a single message in all the application) with the convenience of not having to manage sidekiq or celery for background jobs (they're in the language) and autoscaling.

</comment></comment></comment><comment><USER31> Can we stop repeating the meme that Node.JS is not multi-threaded?

True, your JS code all runs in a single thread, but all the heavy lifting is behind asynchronous interfaces, delegating the work to a threadpool in the background.

What a Node.JS application really is, is a supervisor thread (your JS code) giving work for a bunch of worker threads. Doesn't sound very single threaded to me.

<comment><USER32> Can you start a thread with a JS loop in node so you can have two loops running at the same time doing CPU-heavy tasks?

If not, then calling it multi-threaded is misleading to a programmer who needs such things, and instead you'll need to find a more accurate word.

<comment><USER31> This will repeatedly run 100000 iterations of PKBDF2 in 10 threads:

        process.env.UV_THREADPOOL_SIZE = 10;

        function startOne() {
            const start = Date.now()
            require('crypto').pbkdf2(String(start), 'secretSalt', 100000, 512, 'sha512', (error, result) => {
                console.log(`${result.length}b took ${Date.now() - start}ms`);
                startOne();
            });
        }

       for (let i = 0; i < 10; i++) startOne();

You can use htop or another tool of your choice to confirm this is the case.

<comment><USER32> Are the threads started by way of external process, thread started within a C library, or started with an actual Node.js call i can write directly in node.js code and also use to run literally arbitrary node.js code?

<comment><USER31> If I'm going to assume that by "node.js" code, you actually mean JavaScript code running within the V8 engine as a part of the whole that is Node.JS, then: The answer is no, you cannot do that using pure JS and without using (external) stuff like WebWorkers.

If you are talking more generally about anything we can run within Node.JS, is using Node.JS user-facing APIs and works without doing any modifications to Node.JS, the answer is yes. You can easily achieve that with node's native module support, and I would encourage you to do so over committing the folly of doing anything CPU intensive in JS.

That is if you manage to find any CPU intensive task that isn't already handled by node built-ins or some library out there.

 _Edit:_

Now you may say: "Ahah! You can't use threads using [limited usage of node]. Also threads must be usable for arbitrary workloads in order to consider a thing multi-threaded. Node.js is single-threaded!"

I just can't argue with that. It's just a matter of opinions now about what makes a thing/environment single-threaded, and what makes it multi-threaded.

To me it's simple: [x] Thing uses multiple threads to perform work.</comment></comment></comment></comment></comment>