Show HN: FunctorFlow – An attempt to re-imagine Python libraries - <AUTHOR> https://functorflow.org/ <comment><USER2> The source for the primary package is not published, afaik. The primary entrypoint function is implemented in cython, resisting casual analysis. The stated purpose is to pull (unversioned!) code from the internet and run it, but no mechanism to trust or verify that code is even mentioned.

The advisability of the idea aside, the presentation here feels exceedingly and unnecessarily risky

<comment><AUTHOR> Friend, (1) there is a versioning mechanism already in place (2) Cython is used to optimize the speed (3) FunctorFlow only works on Repl.it. We are in alpha. It does not work on your machine. Just go and install it. See if it works. It doesn't!

<comment><AUTHOR> Lemme repeat once again:

FunctorFlow only works on Repl.it, not on your machine!

Currently making tutorials and assignments for Repl.it community

<comment><USER2> Ah - you might want to clarify that point then, the marketing copy on your site implies that is (will be) usable in any python environment.

<comment><AUTHOR> Yes friend, it _will_ be. Now it is not (while we are in alpha stage).

</comment></comment></comment></comment></comment><comment><USER3> Hard no. My visceral reactions to this:

\- Strings are not names. Names help tooling, strings make tooling harder.

\- A function which takes a string to switch on where behavior radically differs between cases is an anti-pattern.

\- Imports should be explicit and well organized, not magic and scattered throughout the code.

</comment><comment><USER4> I'm all for improving Python, but as feedback on the minimal website, the impression this gives is you're working on the npmification of Python libraries. (I don't mean that as a _good_ thing.)

</comment><comment><USER5> > from functorflow import f, ff

Please don't give these functions simple names like "f". I often use these names as generic function names locally. I know I can use "import ... as" but doing so shouldn't be necessary and will result in everybody using different names, which will hurt consistency, readability and searchability of code.

</comment><comment><USER6> I was hoping this would combine the best features of existing package managers, but instead this seems to have been made in a vacuum.

While making a good package manager/community around it is hard, there's plenty of hard earned knowledge that's been completely ignored here.

This seems to make dependency management harder (imports spilled all over the code base, versioning seems non-existent, documentation for this/plugins? Twitter is not documentation).

How does plugin authoring work? How does distribution and trust work?

What problem is this supposed to solve that isn't addressed by other package managers (or what does this do better than said package managers)

<comment><AUTHOR> Hi friend, This is a humble beginning. Now docs are in YAML-like format and soon they will be published in a readable and interactive format on our new website. For now, I just wanted to share it earlier and to involve other developers in the process.

Thanks for understanding. ;)

</comment></comment><comment><USER7> Where are the docs? The "docs" link just forwards to a Twitter account.

How would one use this with existing Python packages? Or is this a parallel universe to PIP where libraries need to be written explicitly with FF?

The website doesn't mention how FF packages are versioned or how version restrictions between dependencies are resolved.

I see this as quickly becoming full of "left-pad" sized packages, can you analyze a full project to fetch everything ahead of time?

<comment><AUTHOR> Hi friend, This is a humble beginning. Now docs are in YAML-like format and soon they will be published in readable/interactive format on our new website. For now, I just wanted to share it earlier and to involve other developers in the process.

Thanks for understanding. ;)

</comment></comment><comment><USER8> This is just an awful idea all the way through. I'm baffled by how much effort had been put into this. It's almost as if the author went off grid and made this without ever asking anyone if it was a good idea without a second thought for whether it was needed at all. It's also obvious to anyone that you cannot just magically make version numbers disappear and expect everything to go well.

<comment><AUTHOR> Friend, there is a versioning mechanism. I'll share details in our Discord group. Please, subscribe to our email list to participate in our alpha.

</comment></comment><comment><USER9> What's the target audience?

Looks convenient for notebook/repl style use (assuming the the editing environment still supports autocomplete/correct and other niceties). I might just be stodgy and change averse, but I'd be a bit less willing to try things like this in production.

Landing page or FAQ could possibly benefit from an explanation along these lines.

<comment><AUTHOR> Hi friend,

Yes, do not try this in production yet.

Currently, this library works only on Repl.it for the exact reason.

This is why you see the headline with "an attempt". :)

<comment><USER9> I get that. I mean even the idea. Say in 5 years this is far more mature and stable, what use cases do you see it filling? Would they include production applications?

</comment></comment></comment><comment><USER10> Yeah, this is pretty awful. I don't see myself ever using this.

1\. What's the difference between this and just packaging a function as a pip package? The only one I see is just shaving off a few seconds and one less command.

2\. Where is the code running from for each plugin? Eval'ing random code just seems like a bad idea. You can argue that it's the same as blindly installing a pip package, but this apparently happens each time on runtime and I see no option to check the code beforehand.

3\. f and ff are pretty bad function names. I see what you're doing, but why?

4\. Dep management issues aren't as common as you think they are.

5\. Lots of these plugins are already in the standard library. XML to JSON is already possible quite trivially with a pip package. Not only that, but this seems like a less flexible version of that - and more obscure. (how to map xml to json?)

I'm giving this a hard pass for me, but looking forward to seeing my doubts cleared. Cheers.

</comment><comment><USER11> I find how offended some of the people commenting are really compelling. What is the rationale behind this? Like why do it, then why again?

I'm still open to the possibility this could be brilliant.

<comment><AUTHOR> Indeed. Thank you, friend!

I'll keep you updated.

<comment><USER11> How about in the thread? It would be really helpful.

<comment><AUTHOR> Yes sure, in the thread!

</comment></comment></comment></comment><comment><USER12> I find the naming of this library confusing and almost clickbait. I don't see any kind of Functor (object?) in the API, furthermore what the heck is "flowing"? Seems like a copycat to "TensorFlow".

A better suited name which encodes the core feature, "installationless helper routines", could really help this library to become actually recognized for what it can provide.

Here's another comment about the API: I would prefer a syntax like

       f.name_of_function(args)

instead of

       f("name-of-function", args)

The syntactic sugar variant is easy to obtain with Python and has the advantage of much better readability.

</comment><comment><USER13> For those developers who wish to earn an extra income in their free time while helping MOOC students complete their MOOC courses consider registering as an expert tutor here: [https://moocteas-stage- web.azurewebsites.net/](https://moocteas-stage-web.azurewebsites.net/)

Helping others learn while rewarded for doing so.

</comment><comment><USER14> Lack of versioning support makes this absolutely unusable for anything but sandboxing. And since the imports are scattered all over the code, it's hard for security scanning tools to check for vulnerabilities and to update libraries regularly. Pip isn't perfect but at least piprot helps me keep libs up to date.

This tool was clearly designed in a vacuum.

<comment><AUTHOR> Friend, there is a versioning mechanism. I'll share details in our Discord group. Please, subscribe to our email list to participate in our alpha.

<comment><USER14> And how do you expect people to learn that when your website doesn't mention it and your "docs" link is a Twitter account with XKCD comics?

This tool is clearly not ready for any use outside of a sandbox or toy.

</comment></comment></comment><comment><AUTHOR> Hi HN! Meet FunctorFlow ^_^

Think of it like lego blocks for your next tech-toys and hopefully (in the future) for your next big-thing ^_^.

If you love tinkering with new toys then give it a try with Repl.it: [https://dev.to/t/functorflow](https://dev.to/t/functorflow).

If you want to meet cool kids like you then join our club at Discord by subscribing to our email-list.

...and follow updates at [https://twitter.com/functorflow](https://twitter.com/functorflow)

Together, we will re-imagine and re-invent how Python apps should be developed.

Lemme know what you think!

Have fun!

\- Sam

<comment><USER15> "Lego blocks" "tech toys" "tinkering" "cool kids" "next big-thing" \-- perhaps they are your hopes for this project, but these are all vague, filler words that don't tell me what this thing actually does. If I take a look at it what do I see?

It's not until I get to the twitter account that I see that it looks like you're trying to create not just a different python package manager, but a new ecosystem that's not backwards compatible. I don't think this is a good idea. The Python package ecosystem is gigantic and one of the largest, most mature language ecosystems out there. If you're trying to make the way Python apps are developed better, try taking a look at alternate Python package managers like `pipenv`, `poetry` and `anaconda` and figure out what you like and dislike about them. Try making changes to them before you reinvent the wheel.

As it stands right now, this isn't even a wheel. It's an uninspired marketing page on twitter that is not doing a great job of convincing anyone it's better than the alternatives. In fact, it's not even clear that you're even aware of the alternatives.

<comment><AUTHOR> [https://twitter.com/functorflow/status/1157396077043167232](https://twitter.com/functorflow/status/1157396077043167232)

<comment><USER16> That diagrams is 90% about different versions of PYTHON and not python modules. So I have no idea what it has to do with the discussion at hand.

Even if it was about modules, if anything FunctorFlow makes it worse because you now have X+1 things creating modules (FunctorFlow has to live somewhere after all so it doesn't remove the previous system).

</comment></comment></comment></comment><comment><USER17> Well, this has all the signs of well-meaning but inexperienced developers seeing a market opportunity without appreciating the many difficult problems that will have to be solved to produce something that is secure, reliable and interoperable with current development ecosystems.

Rather than abuse them, we might perhaps use them as an example of the Dunning-Kruger effect.

</comment><comment><USER16> So, aside from other comments, doesn't this break the already mediocre tab completion side of python? Scrolling through API docs to figure out what exactly that method or argument was called gets really tedious after the 5000th time.

</comment><comment><USER18> Why?

</comment><comment><USER19> Absolutely terrible. I can see why data science types who use / produce a lot of (typically throwaway) Python code might be enamored with this, but as far as the Python language goes, functorflow will be another nail in its coffin if it ever gains mass-acceptance. It will turn the already messy and inconsistent ecosystem into shambles that no sane person will want to build anything robust on top of.

With the mass migration of Python developers into Go, this is the last thing that Python needs.

<comment><USER20> > I can see why data science types who use / produce a lot of (typically > throwaway) Python code might be enamored with this

I think that's honestly its killer app. I don't do data science, but I often use jupyter to write short prototypes, and make plots and such.

I've been looking for something that lets me:

1\. Set up a jupyter notebook.

2\. Run some tests or experiments, pulling in libraries as needed.

3\. Ignore it for a few years.

4\. Come back and run the same notebook without everything breaking.

If this could patch into Python's native import mechanism and let me specify the repo version as todays date, it'd be great.

        import functorflow
        functorflow.repo("2019-08-04")

        from ff.package_name.whatever import foo

If that gets you a replicable set of versions, it'd be pretty great.

<comment><USER16> Hmmm, that feels more like a jupyter extension than a standalone module. The extension could intercept imports automatically without the need for a prefix. Might require a custom kernel that does some magical per notebook lib mapping. Maybe it installs libs in version specific central folders and then does some symlinks to build a notebook specific lib folder based on the date specified. Feels more seamless but also more complicated.

<comment><USER21> I'd actually implement it as a transparent venv and use pinned versions in a requirements.txt – with hashes, of course.

<comment><USER16> That'd make sense. You'd need to pin all transitive dependencies as well and cache the venv. Not sure if you'd need to find older versions of new direct dependencies to avoid conflicts. For example, you run the sheet in 2015 and then again in 2019 but with a new import. That new import's latest version has a transitive dependency that you already pinned but has different version requirements than what you pinned. I wonder if you can embed the requirements.txt inside the sheet itself.

Only issue is that it'd be really space inefficient, my DS venvs clock in at 400+mb each so having one per sheet will probably quickly become unusable. Which is why I thought of some sort of smart system wide caching akin to maven/ivy. But I'd forgotten how complicated python dependencies (binaries, c code, etc.) were and how little api support pip had.

<comment><USER20> > You'd need to pin all transitive dependencies as well and cache the venv.

I think to do this stuff, you need hooks in Jupyter to setup and teardown the venv before it runs the kernel. (And generally Jupyter would want to clean up unused venvs to mitigate the teardown hook not firing.)

> Only issue is that it'd be really space inefficient

A venv is overkill. You can just run the kernel in a regular directory, and `pip install --target kernel_dir foo-bar-pkg` to put packages directly in it. As long as the linker sees it, third-party libraries will work. This technique is used in the serverless project[1] to bundle dependencies for use on AWS Lambda.

> Not sure if you'd need to find older versions of new direct dependencies to > avoid conflicts.

Curation is a solution to this. Stackage[1] is popular in the Haskell community; they build a consistent version set of everything every night and curate stable releases periodically.

With curation, a date and a set of top-level packages is enough to pin your dependencies.

[1]: [https://github.com/commercialhaskell/stackage#frequently- ask...](https://github.com/commercialhaskell/stackage#frequently-asked- questions) [2]: [https://github.com/UnitedIncome/serverless-python- requiremen...](https://github.com/UnitedIncome/serverless-python-requirements)</comment></comment></comment></comment></comment></comment>


How I write back ends - <AUTHOR> https://github.com/fpereiro/backendlore <comment><USER3> A great write-up. Two interesting points:

1) No Ansible. No Kubernetes, nor containerisation. Starting on a single server/vm and controlling deploys with a bash script.

2) An interesting take on microservices: "One server, one service... 2-3k of code can hardly be called a monolith.... if two pieces of information are dependent on each other, they should belong to a single server."

As a (mainly) backend dev who does like devops and microservices, I agree on both points.

KISS - there's no need to start off with over-complicating everything for no reason. When there's a need for Ansible, introduce it at that point. When you think you need microservices, think about it a lot and move to them if you really do require them.

But certainly don't start off using a full Ansible + Kubernetes + Docker + microservices stack. You'll spend all your time managing these, and relatively little time actually being productive.

<comment><USER4> > You'll spend all your time managing these, and relatively little time > actually being productive.

I see this a lot, and I'm always somewhat surprised by it. It took a couple of days to put together our kops-based cluster on EC2, wire it in to Gitlab CI and get the various load balancers, ingresses et al plumbed in, and then perhaps an hour a week maintaining it from there?

I do agree it's not the right place to start, but I think the standard operational complexity of k8s is frequently overstated. The argument I _do_ buy against k8s is that when it goes wrong, debugging can be a nightmare.

<comment><AUTHOR> I fully concur with your last statement. I find that the main advantage of having less moving parts is that there's exponentially less things that can go wrong - and that makes reliability possible with very small teams. The setup cost is usually not steep when compared to the overall lifecycle of a project, I fully agree.

The other argument I have against many DevOps tools (and most software tools in general) is that the patterns of the solutions they provide don't generally match the basic patterns of the problems they are solving - the consequence is the sheer complexity of understanding the tool, especially when something goes wrong. In the past, this hasn't only costed me time, but a lot of frustration that I'd rather avoid - even at the price of having to build my own solutions.

<comment><USER4> > I find that the main advantage of having less moving parts is that there's > exponentially less things that can go wrong - and that makes reliability > possible with very small teams

Right but it is a trade off to some degree. For us, to have ephemeral feature branches deployed to QA completely in a hands-off manner means we can do multiple releases per day as a small team. Most (almost all!) of the advantages we have seen from moving to a k8s infra have been DX, rather than UX.

<comment><USER5> What is your workflow enabling this?

<comment><USER4> Nothing esoteric, git trunk with feature branches that build, test and deploy in Gitlab CI to _a-feature-name.our-company-name.local_ on k8s, notifies QA team that there's something to check, gets approved, merged to master and released automatically, functionally tested on canary prod deploy and then rolled out wider.

</comment></comment></comment></comment></comment></comment><comment><USER6> Awesome writeup.

You mentioned you're still "finding an elegant way to generate, use and clean up CSRF tokens".... Consider implementing an "httponly", "samesite=strict" cookie for security check. Apple recently got on board so all major browsers are supporting this. One goal of samesite is to make CSRF tokens redundant.

We roll out at least three different cookies with varying expiration times. Including each of "strict", "lax", and "none". They are there for various reasons.

Of course additional checks occur, especially for big operations, but we no longer keep track of multiple CSRF tokens for each user/device/browser combo.

Keeping track of CSRF tokens via JavaScript, considering multiple tabs, is kind of a nightmare... and it's never as secure as you initially think.

<comment><AUTHOR> Thank you for your suggestion about using samesite=strict! I'll look into it a bit more - the only downside I see is support for older browsers, which is a personal vice I can't kick. If I see that there's no elegant CSRF solution, this will probably be my fire escape. Thank you for that!

I added more info in the document regarding cookie/session management, particularly its expiry ([https://github.com/fpereiro/backendlore#identity-- security](https://github.com/fpereiro/backendlore#identity--security)). The approach I take is to make cookies last until the server says so. I'm very curious as to how you use cookies with varying expiration times and would love to know more if you have the time.

<comment><USER7> Seconding @fpereio's request - I too would like to know more. CSRF handling is painful, so your approach is very interesting, but I don't understand it fully yet.

<comment><USER6> Until several years ago cookies couldn't be used for CSRF tokens, because they were sent with every request to your site, regardless of where the request originated. SameSite changes that because you can now control when a given cookie is sent with a given request, based on how a link was followed. It puts the work on the browser.

SameSite is just a property you use when setting a cookie, like the expiration, or url path. There are three settable values: None, Lax, and Strict.

'None' sends the cookie with every request, like how browsers "historically" acted.

'Lax' is the new default [1]. If someone follows a link to your page, these cookie are sent with the request. If a third-party site POSTs to one of your pages, these cookies won't be sent.

'Strict' cookies will never be sent to your server if the request originated from a third party, even if it's as innocent as following a link from a search engine.

So use these types to your liking. Your needs vary. If you want the user to appear logged in when they follow a link to your site you're going to use None or (most likely) Lax.

As far as CSRF, you can generate a single CSRF token at login and set it in a cookie as with SameSite=Strict. Regenerate it every so often, or at "big" events, and voila. One CSRF token per device session. No worries about multiple tabs open, and you have reasonable expectation to only have to keep track of a single token at a time. The kicker is, you technically don't even have to have a CSRF token if you don't want. If you're serious about nobody linking to an "internal" authenticated page, just set the primary session id cookie as strict. (I would personally do more than this, but hey.)

I do secure portals. So we set several cookies across the board. Varying expiration times. They all have their purposes.

 _[1] While you can consider Lax as the new default, if you omit the SameSite property on a cookie it is actually attributed as something referred to as 'NotSet', which is unsettable, and acts a hybrid between Lax and None. Browsers vary a little here, but it's only to maintain reasonable backwards compatibility at the moment, and both Mozilla and Chrome developers plan to remove this hybrid functionality in the future. Basically.. a cookie will act like None for 2 minutes, then act like Lax after that, etc._

<comment><AUTHOR> Beautiful answer, thank you!

Since I want to support as many browsers as possible, I'm going to still use CSRF tokens, but a single token per session as you suggested. I just outlined the approach here: [https://news.ycombinator.com/item?id=22268152](https://news.ycombinator.com/item?id=22268152) .

Thanks for your detailed feedback, it's very valuable to me.

</comment></comment></comment></comment></comment><comment><USER8> > _Speed - S3 is not very fast, even accounting for network delays. Latencies > of 20-50ms are par for the course_

I've used S3 in past projects. These numbers are in line with my experience. And frankly, I'm good with them. S3 is dirt cheap compared to EBS, and it IMO doesn't generate a lot of production incidents (compared to, say, AWS RDS MySQL). The latency is on the high side compared to some things, but if the user is doing an "upload" operation of some sort, 50ms is tolerable, IMO. And it always seems to be _way_ more efficient that just about everything else, in particular, it's usually next to some Mongo/SQL DB that is stuggling under the load of feature creep. S3 has basically never been on my bottleneck radar.

The biggest thing I find people doing wrong, particularly in Python, is a. failing to keep the connection open (and so paying the TCP & TLS handshakes repeatedly when they don't need to, which adds a _significant_ latency hit) and b. making boto calls they don't need to, such as "get_bucket" which in boto will check for the bucket's existence. You almost always can guarantee the bucket exists as a precondition of normal system operation… and the next call to GET/PUT object will fail anyways if something is amiss.

<comment><AUTHOR> Fully agree. 20-50ms for something like S3 is fantastic and proof that we live in the future. S3 is only slow when compared to interacting with redis or a FS layer powered by node (by about an order of magnitude). That's why I feel that complementing it with a self-managed FS can work well in certain settings. And interestingly enough, because S3 is so solid, you don't have to be so paranoid with your own FS layer, since S3 is there to bail you out in case of data loss.

Regarding S3 pricing, ~0.09 USD per downloaded GB could potentially become a significant bill (~90 USD per TB). I'm building a service that serves pictures and limiting the financial downside is very important for the economics of the business. More than the actual magnitude, the certainty that costs are a linear and predictable function of storage makes me sleep better.

An implicit point that I just realized by re-reading your comment: if you have a significant amount of data (let's say, > 100GB) a local FS is only cheaper if you're using instances or dedicated servers with large disks - and probably this is prohibitively expensive in AWS (EBS). I'm particularly considering Hetzner dedicated servers with large disks.

<comment><USER5> Can you break down the financials for us?

<comment><AUTHOR> At some point I'll do a public writeup of my planned cost structure. Hetzner's EX62 servers ([https://www.hetzner.com/dedicated- rootserver](https://www.hetzner.com/dedicated-rootserver)) are about 10 USD/mo/TB of disk and it would be hard to go over their bandwidth limits . Even allowing for RAID and/or multiple locations/servers for a file node, the cost is still low. The main advantage in terms of cost is the possibility of not spending money on outgoing bandwidth.

<comment><USER9> Hetzner's pricing for dedicated servers is impressive. However, I'm curious about how do you deal with (or whether you are worried about) the lack of their presence in North America, Asia, Oceania? Unless most of your customers are in Europe, latency might be an issue ... What is your experience in this regard?

<comment><AUTHOR> That's a good question. I'm going to try first with Hetzner and measure latencies from different places; if this becomes an issue, I'll definitely consider complementing the infrastructure from elsewhere. But I'd rather start simple.

I currently run a web service from North American servers and the performance here in Europe is quite good. I think this experience - very, very limited - gives me confidence that the wires across the world are good enough, provided your data center is close to the main pipes.

<comment><USER9> All right, good luck! Please share your Hetzner experiences here - I will be watching this thread.

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> I really love articles like this. There's so much value in quickly summarizing years of experience with proven tools.

This writeup in particular is great because it starts with simple architectures and progresses to horizontal scaling with a load balancer -- it really is 75% of what you'd need to actually implement a production-ready application backend, with clear links to get the other 25%.

<comment><USER11> +1 and I'm sure the article could inspire others to do the same with other stacks. I like how it compares to other similar stack where OS is CentOS, serverside lang is just PHP and some other layers added, like HAProxy and/or Varnish cache, or even Apache or Lighttpd instead of the omnipresent Nginx for the webserver

</comment></comment><comment><AUTHOR> Wow. I wasn't expecting this level of interest, not in a million years. I'm very grateful (and a bit overwhelmed!). Thank you for your thoughtful comments. I just added a few more clarifications to the document, mostly inspired by your interactions.

<comment><USER12> Your article showed up at exactly the right time for me :). A project I'm working on has been increasing in 'devops' complexity, and I'd been doing some research into Docker/Ansible/etc. because I felt that perhaps my simpler solutions weren't ideal.

After some research I concluded my approach was still preferable, at least for now, but I still felt some unease because everyone seems to be using Docker and whatnot. Your article really helped me feel more confident in my choices. Thanks!

<comment><USER5> Based on your analysis, how did you decide not to move to containers?

<comment><USER12> Argh. long comment got deleted. So, short version:

1\. I already know how nginx/apache/linux work, so it's been pretty easy to spin up a new VPS and configure it to handle potentially multiple apps, rather than learning how to work with Docker or the like.

2\. I run pretty much the same stack everywhere (Phoenix/Elixir) and I don't foresee running into issues where I need to run multiple versions of, say, elixir or node.

3\. For deployment I quite like git + hooks to handle things. and because of some of Elixir's particulars, it's real easy to deploy a new version and recompile without having to completely restart the app.

4\. I don't like the idea of adding another layer of complexity. The way I see it, if anyone else needs to work with me on the 'devops' part, they better know how to work with linux/nginx anyways. And if they already know, they can pick things up pretty quickly. Knowing how to use Docker would just add another thing for them to learn, another thing for me to keep an eye on, and another thing to follow updates and security issues on.

5\. I'm very much of the article's school of thought that scaling stuff on a single server for quite a long time is possible. Perhaps especially with Phoenix/Elixir. So at worst I could see myself moving an app to a separate VPS, but I don't need to 'orchestrate' things and whatnot.

6\. I haven't properly researched this, but I suspect that, for quite a few of our clients, running things on a separate server is a legal requirement. A container would not only be an alien concept to them, but it might very well just not be allowed (would love to hear input on this though).

tl;dr: I just don't need it. I can spin up a server and get everything set up in < 30 mins, some of it done manually and some of it with some simple bash scripts. and I can't think of a very good reason to learn and implement another layer that sits in between my server and the running app(s).

All that said, I don't know what I don't know, so I'd really love to hear where perhaps I might benefit from using Docker/Ansible or the like! I'm not at all against these tools or anything.

EDIT: I'll add that I do think perhaps Docker might be useful for local development, especially when we start hiring other devs. Am I correct in assuming that's one of the use cases?

<comment><USER13> > EDIT: I'll add that I do think perhaps Docker might be useful for local > development, especially when we start hiring other devs. Am I correct in > assuming that's one of the use cases?

As I understand it, something like that is supposed to be one of the big draws: Since everything runs in a container the environment is the same regardless of the system packages, so you shouldn't have any "works on my machine" bugs.

That said, we had a session recently where a system that was just converted to docker was being handed off from development to maintenance, that also acted as a crash course for anyone interested in docker on the other development teams. I think only a fraction of us actually got it working during that session, the rest having various different issues, mostly with docker itself rather than the converted system.

<comment><USER12> Interesting. Thankfully most of my current work involves a stack that works fine on Linux/Mac, and I don't foresee many other devs needing to work with it, least of all devs using Windows.

But I recall doing contract work where it took me and every new developer literally a day (at least) to get their Ruby on Rails stack working with the codebase, and where we often ran into issues getting multiple projects running alongside on account of them needing different versions of gems or Ruby. I can see how Docker would be a great timesaver there.

<comment><USER14> Are you using ASDF? I generally commit a .tool-versions file to source control for every project. It specifies the versions of each language (usually Erlang/Elixir/Node) to be used for a given project.

I started using ASDF for Elixir projects, but over time I've basically replaced RVM, NVM and all other similar tools with it.

<comment><USER12> Back in the RoR days I guess it was RVM that I used. I've also used NVM in the past.

But honestly in my current work I've not needed it yet. As far as Elixir/Phoenix goes, things have been stable enough so far that I've not needed to run multiple versions, and I try to rely on node-stuff as little as possible (more and more LiveView), so I can get away with either updating everything at once, or leaving things be.

Furthermore, when various apps diverge too much, I usually find there's a need to run them on separate VPS', which I find cleaner and safer than trying to make them run alongside each other.

But thanks for reminding me about ASDF. It seems like a much nicer solution than using multiple tools!

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> Awesome writeup, I agree with mostly everything. A few small tips:

* Use `npm ci` instead of `npm i --no-save`. This gives you the exact packages from the lockfile, which is more reliable.

* You might like systemd .service files instead of `mon`. It's available by default on ubuntu LTS, and you're down a dependency that way. The .service files can be really simple: [https://techoverflow.net/2019/03/11/simple-online-systemd-se...](https://techoverflow.net/2019/03/11/simple-online-systemd-service-generator/)

<comment><AUTHOR> Thanks for your comment!

\- I don't use lockfiles - instead I use very very few dependencies, and I always target a specific version of each of them. I know this is not the same as a shrinkwrap, but I'm crazy enough to feel that package-lock.json adds visual bloat to my repo.

\- I am biased towards mon because it's probably less susceptible to changes over time than Ubuntu itself, and because it can be installed with a single command. But systemd definitely is a good option too if you're using Ubuntu!

<comment><USER16> The main problem with your approach is that you have full control over your dependencies but no control over _their_ dependencies, unless you're inspecting every single package in your dependency graph to make sure that every one of them targets specific versions in its package.json. Realistically, using package-lock.json and npm ci is the only way to ensure that everyone on your team and your deployments are all working off of the same node_modules folder.

</comment></comment></comment><comment><USER17> If you replace NodeJS with Django and Redis with Postgres it pretty much maps 1:1 with how I deal with it.

Once/if postgres becomes a bottleneck I simply add redis to cache, the next step is doing some stuff asynch with RabbitMQ + Celery and you can scale really really really far...

<comment><USER18> To many people's surprise, I can crank out a fully-functional side project in two to five days (latest ones were [https://imgz.org](https://imgz.org) and [https://nobsgames.stavros.io](https://nobsgames.stavros.io)) by having already completely automated devops.

Much like you, I have a Django template project and use Dokku for deployment, so I can use the template to create a new project, commit, push, and it's already up on production.

Here's mine:

[https://github.com/skorokithakis/django-project- template](https://github.com/skorokithakis/django-project-template)

Having all the best practices and boilerplate already sorted helps immensely.

<comment><USER19> I've worked a lot on this sort of thing in my company, as we spawn new projects every few weeks. We went from 4 days to deploy a fully-functioning Django+React+CI on AWS to literally 30min (much of it just waiting) using similar approaches.

And now we have the same thing for API Platform, NestJS, Vue, NextJS... Even React-Native with Fastlane and CodePush :) It's been invaluable, and a very strong commercial argument

<comment><USER5> The situation isn't clear to me. What changed that lead to more productive deployments?

<comment><USER19> Using templates, boilerplates and generators.

Creating a new project is simply "make generate" from our generator, then answering a bunch of questions about what tech stack you need. This works mostly with templating (using Plop) and a sprinkle of automation for shell tasks (eg `pipenv install`)

<comment><USER5> Thanks for clarifying. What used to take 4 days to set up a new server can now be done in <1hr. How often is that being taken advantage of? "Any time we need a new server" is determined by a policy determining whether to bolt onto additional servers or not. Curious about that policy..

</comment></comment></comment></comment></comment></comment><comment><USER20> Is there a specific reason you haven't replaced the Vagrant stuff with a container-based approach? You can also do stuff like using the lxc backend[0] for Vagrant if that's more your speed. While containers aren't strictly better, they are certainly lighter weight and quicker to spin up/tear down, though performance characteristics and OS settings/etc can definitely differ.

I've found linux containers to be fantastic for both replicating pieces of the environment and making your e2e tests a little more offical. Running postgres itself (migrating it like you would your real app, inserting data like the app would, then tearing it all down in seconds), apps like mailcatcher[1][2] and actually letting your app send mail (this way you don't even need a local/log- only implementation), running redis locally, and replicating S3 with a project like minio[3] is very easy and fast.

You can take this even further also by building a way to completely spin up your production environment to a certain point in time locally -- though all the resources for the ecosystem of apps may not be on the same VM image (otherwise you could just snapshot and technically go to the point in time), if you can assemble the state and the backups, you can hook them all up to the containers running locally (or in a VM running locally to be fair) and get an approximation of your entire system.

Of course, it is somewhat less true-to-production (given that production is you running a VM), but you can remove that barrier with things like linuxkit[4].

[0]: [https://github.com/fgrehm/vagrant- lxc](https://github.com/fgrehm/vagrant-lxc)

[1]: [https://mailcatcher.me](https://mailcatcher.me)

[2]: [https://github.com/sj26/mailcatcher](https://github.com/sj26/mailcatcher)

[3]: [https://www.minio.io/](https://www.minio.io/)

[4]: [https://github.com/linuxkit](https://github.com/linuxkit)

<comment><AUTHOR> You raise interesting points!

In general, I don't use Vagrant. I simply use my host OS (Ubuntu) and my local node & redis. There's usually no further moving parts than these, so it is easy to achieve development in an environment that is practically equivalent to the Ubuntu servers/instances in the cloud, the main difference being a different folder path, different files stored and different info on the DB.

When running the app locally, I still use services like S3 and SES - <USER2> I might not send emails, to keep things as "real" as possible.

I do use Vagrant when the project requires installation of other tools beyond my basic toolkit (like MongoDB or Elasticsearch) - in this case, I want that software to stay within the VM.

In general, I'd rather avoid virtualization because it represents an extra layer, but if it is necessary for idempotence or having a dev environment that mimics the prod environment, I'd definitely embrace it.

<comment><USER20> I'd really recommend getting more intuition/familiar with containerization and introducing it -- it is an essential part of the modern toolkit and really isn't very hard to use these days. It will absolutely introduce/encourage idempotence and bring your dev environment just a little closer to your prod environment. Modern container runtimes can even run rootless containers, swap out the "virtualization" engine underneath (as in you can even run your container in QEMU without changing it for more isolation), so it can be a net positive for security too. There's also no need to swap out folder paths and file locations -- just moving around and disconnecting/reconnecting your data.

Containerization is actually not virualization (which is why it was in quotes ealrier), they're basically better-sandboxed local processes (just like your local redis instance), and the better sandboxing has benefits for both development and production. So if you squint, it's actually very similar to just running redis yourself on a similar OS -- just now when you shut it down you don't have to manage any folders.

Of course, use what works -- there's no need to fix things that aren't broken, but there is a reason that the defacto deployment artifact of the modern application is very quickly becoming a container (if it isn't already) -- in relatively trusted environments they give just the right amount of isolation and reproducibility, and centralized management (systemd even has space for containers via systemd-nspawn[0]).

[EDIT] - somewhat unrelated but if you haven't I'd _really_ suggest you give GitLab a try for your source code management -- not only does it come with a free to use docker image registry, it comes with a fantastically easy CI platform which can make automated testing easier. You get 2000 free minutes a month, and actually if you bring your own hardware (which is as easy as running a container on your local machine and pointing it at gitlab.com to run your jobs for you) it's all free. There are lots of other integrations and benefits (kanban style issue boards, gitlab pages, wiki, etc), but it's one of my gotos for small projects and I introduce it to corporate clients every chance I get. Microsoft-owned Github is doing it's best to compete now that it has endless pockets, but GitLab has been offering an amazing amount of value for free for a long time now.

[0]: [https://www.freedesktop.org/software/systemd/man/systemd- nsp...](https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html)

</comment></comment></comment><comment><USER21> While this is a fantastic resource, I feel like it's indicative of a larger problem: why is all of this still needed ? Is there no way to abstract all of this and focus on the actual functionality of the webservice ?

This is probably the answer that Google App Engine, by providing all the bricks needed to replace Redis or the FS or S3, and all you need is to write your own application code. All the provisioning and monitoring is taken care of. Is there an open alternative to it I could run locally or on my own server ?

<comment><USER22> Depending on what you're building, you can even go with [https://hasura.io/](https://hasura.io/) and focus on the user-facing parts of your application! It still makes it reasonably straightforward to implement custom logic, and you only need to write backend code that directly benefits your domain, skipping all the CRUD plumbing.

<comment><USER21> Hasura seems to be very specifically about a database and its API. What I'm looking for is an environment that is more focused on application code, and which gives you the API to communicate with the external world. That environment would provide all that is given in the article by default.

</comment></comment></comment><comment><USER23> If you're going to go the redis route Amazon has a managed service: [https://aws.amazon.com/elasticache/](https://aws.amazon.com/elasticache/)

It shouldn't be to expensive and can save you a lot of time in setup.

I think I'd also recommend using ansible instead of plain SSH. It has a lot of stuff built in and for simple deployments shouldn't be too hard to pick up.

Future you will probably appreciate an off-the-shelf solution, instead of something custom :).

It is nice seeing something like this put together though. There are so many moving pieces in a project it can lead to decision making paralysis where you find yourself going down rabbit holes, second guessing everything you do and seemingly never getting ahead. At least that's what happens to me when I'm doing something new.

<comment><AUTHOR> Wow, I didn't know that AWS had a managed redis service! I thought Elasticache was only for Elasticsearch. Thank you for pointing this out.

In my experience, setting up redis is very easy and not time-consuming; what is not trivial is to store large amounts of data in it, which requires either getting more RAM or considering a clustering option.

I'll very much have this option in mind for upcoming projects where I'm not the sole owner.

Regarding Ansible vs bash/ssh, I'll probably (hopefully!) come to the point where I'll need something more sophisticated. Ansible is definitely my benchmark for an elegant solution to idempotent provisioning and infrastructure as code. I might not help writing something custom in js, but I definitely will have to go beyond bash and purely imperative approaches. I'll be sure to share what I learn along the way.

<comment><USER24> Another option over Ansible/Saltstack for managing remote hosts, is using something like Fabric [1]. It's a "remote execution framework"; basically a Python wrapper around SSH for defining hosts and writing tasks that are executed in a shell on your fleet.

[1] [https://www.fabfile.org/](https://www.fabfile.org/)

<comment><AUTHOR> Will take a look at Fabric, thank you!

</comment></comment></comment></comment><comment><USER25> I liked this comment in the article (might use a variation myself):

"If you must sting, please also be nice. But above all, please be accurate."

</comment><comment><USER26> _An HTTP API allows us to serve, with the same code, the needs of a user facing UI (either web or native), an admin and also programmatic access by third parties. We get all three for the price of one._

At a high cost, writing an intermediate JSON API is no free launch. For solo dev and, unless the UI is someone else problem, ask yourself if you really need all this(many frontends, third party access, etc..) before blindly following that advice. Server-rendered-html(with some JS) might be more than enough.

<comment><AUTHOR> You raise a great point.

The web frontends I write always draw views entirely through client-side js, and interact with the server mostly through JSON. The only HTML served is that to bootstrap the page, plus static assets (js & images). If you use server- side rendering, writing an HTTP API might indeed represent more work. And I'm in no position to say that client-rendering is better than server-rendering.

Just added a clarification in this regard in the document. Thanks for bringing this point up.

</comment></comment><comment><USER27> Fantastic article.

Less "how we used fancy schmancy crazy stuff your mom warned you about, and you should too, or feel bad."

More real world truth.

</comment><comment><USER28> Thanks a lot for the article! Loved it. Within my limited experience, I also agreed a lot, which I find encouraging.

A few quick questions for the author if I may:

\- Do you have recommendations for other articles and books or courses like this?

\- What's your take on a few alternatives: Python (rather than node) and Google Cloud or Azure (rather than Amazon)?

\- do you lose sleep worrying about the security of your servers? :-)

Thanks!

<comment><AUTHOR> Glad you liked the article!

I don't have recommendations for other articles like these; most of what I wrote came from personal experience or things I learned directly from other people working with me. I also have learned a lot from writeups by people running web services, especially when they encounter issues and/or suffer downtime or data loss. These real-world lessons can be highly enlightening, particularly if they're written in an honest way - I cannot point to any one in particular, but that's what I would look for, especially for services that run a stack similar to the one you're planning to use.

As for Python vs node, I recommend using the language you love the most, as long as the runtime you use is decently performant. node itself is blazing fast, but I would probably use it anyway if it was 10x as slow. I have no experience running Python as a web service (I've only done some local scripting with it) but I'm sure it can work perfectly well and I know of at least a couple projects which use it successfully at big scales.

I recently read a writeup by DHH about Ruby representing only 15% of the infrastructure cost of Basecamp ([https://m.signalvnoise.com/only-15-of-the- basecamp-operation...](https://m.signalvnoise.com/only-15-of-the-basecamp- operations-budget-is-spent-on-ruby/)). The takeaway here is that your choice of database and architecture will probably determine your costs and performance, not the programming language.

Regarding alternatives to AWS, I haven't taken the time to consider them, honestly. I use AWS very sparingly (with the exception of S3 - which has a track record that no equivalent service offers, if only for the sheer amount of time they've been at it - and SES, which I use only because I'm already using S3 and I might as well just have one cloud provider). For work with clients, they all seem to prefer embracing the dominant vendor and I don't have any powerful arguments to dissuade them from this.

At the moment I'm only responsible for a couple of services with low traffic (and handling no financial transactions nor pacemakers), so I sleep with my phone off. I've been responsible for largish infrastructures and in those cases, the quality of your sleep drastically declines. Soon and hopefully, however, I'll be running a service with a lot of user data and reliability guarantees, so I won't be able to turn off my phone anymore at night. This time, however, I'll be fully in control of all design decisions and I hope that, by systematic treatment and elimination of sources of failure, outages and emergencies should happen asymptotically less as time goes by - and with any luck, the lessons learnt can also help others that are in the same position.

<comment><USER28> Brilliant reply, thank you! I think you just persuaded me to switch to AWS.

Do you do any work with web sockets? I'm a bit worried about that wrt Python, whether it will scale well.

I guess my biggest security concerns are messing up my REST implementation, dependencies in Node being compromised, and vulnerability in servers I'm using, e.g. nginx, redis.

<comment><AUTHOR> I normally don't work with websockets, just JSON, but perhaps that's because all the applications I work on are basically glorified CRUDs. If your application is essentially a CRUD, however, JSON might be more than enough. I've seen thousands of JSON requests/responses per second being handled by small virtual instances running node.

The first thing I'll do as soon as the product I'm building starts bringing in revenue will be to pay a security expert to perform a security audit of it all. In the meantime, I try as much as possible to eliminate backdoors (use the API for everything), simplify the data flows, drastically reduce the amount of dependencies and of moving parts, and as much as possible use technology that's been around for a few years (nginx, node and nginx all make the cut :).

<comment><USER28> As I need to push events/data (collaborative app) I'll need websockets unless I decide to do some insane polling.

Totally with you on the security expert, definitely a priority.

Thanks a lot for taking the time to reply, you've been really helpful!

<comment><AUTHOR> Using websockets for a collaborative app makes all the sense in the world (incidentally, I realize I've never written one an app of that kind).

My pleasure! Being part of this thread has been an amazing learning experience.

</comment></comment></comment></comment></comment></comment><comment><USER29> Good stuff. I love seeing really practical articles that just lay out, "Given my experience and the lessons I've learned, these are a good baseline of general best practices".

</comment><comment><USER30> Regarding Architecture B, I was always curious why use nginx as a proxy to node? In my experience it is perfectly capable as an application server as well (OpenResty in particular), no need for yet another component.

Also, I consider deployment via Docker a sensible default with all the niceties in tooling. For example I find it convenient to set a restart policy on the container process.

<comment><AUTHOR> The main reason I use nginx is because it is _so easy_ to configure HTTPS with LetsEncrypt & Certbot. Perhaps there's an almost equally easy solution now that only uses node, but last time I checked (2018) it was messy.

In this article ([https://medium.com/intrinsic/why-should-i-use-a-reverse- prox...](https://medium.com/intrinsic/why-should-i-use-a-reverse-proxy-if- node-js-is-production-ready-5a079408b2ca)) there's another good reason: by not making node manage the HTTPS certificates, no node library that I use (directly or indirectly through a dependency) can represent an attack surface regarding the certificates. But I must confess this is marginal compared to the ease of installation and maintenance that I already mentioned.

<comment><USER12> Another good reason is that once you start running multiple apps with different tech stacks, it's _so_ much nicer to be able to take care of the config using the regular sites-enabled 'workflow'! I have a server that runs a node app, and a few Phoenix apps, and all I have to do on the nginx end is forward to the correct port (and of course set up https with LetsEncrypt is real easy too).

</comment></comment></comment><comment><USER5> Can anyone recommend resources related to CAP theorem decision making? I want to go further than I have for understanding tradeoffs.

</comment><comment><USER31> Great article in general, but this is _absolute gold_ :

"Before you rush to embrace the microservices paradigm, I offer you the following rule of thumb: if two pieces of information are dependent on each other, they should belong to a single server. In other words, the natural boundaries for a service should be the natural boundaries of its data."

<comment><USER32> I'm curious what's a good rule of thumb for "dependent on each other". A foreign-key relationship? That's... pretty much everything unless you have two completely separate business lines.

<comment><AUTHOR> This is not an easy question to answer, but in my experience it pays off to ponder it carefully.

So far, the heuristic that seems to work better for me is to keep the overall surface of the app small and try to keep everything in a single app. Lately, however, I'm convinced that extracting parts of the code as separate services (in particular, for handling users, logs, and statistics) would make the overall structure simpler. Dissecting this a little bit, an external user service is equivalent to middleware, whereas logs and statistics are write- only, after-the-fact side effects (from the perspective of the app), so bidirectional interaction with them is not needed for the purposes of serving a particular request. What I try to avoid is to have a server route that involves (necessarily) asynchronous logic between services to check the consistency of an operation.

In terms of "business logic", I feel that whatever lies within a transactional boundary should be in the same service; i.e., if two users share a resource, both the information of who owns the resource and who can access the resource should be in the same service. But I suspect there might be a lot more of thinking to be done in this regard.

</comment></comment></comment><comment><USER33> Thanks for the writeup - I feel like trying redis again just because it opens some interesting possibilities (yes yes could be done in SQL whatever).

Regarding "ulog" in your example application, do you keep those lists unbounded? Does this work well in practice?

<comment><AUTHOR> I am consdering to move "ulogs" out of redis and into files because of the potential memory usage - perhaps a hybrid approach where only new logs are in redis would work best. I don't have a definitive solution yet. However, this would be a problem when handling tens of thousands of users or more, which makes it a problem I'd love to have!

I'm also considering letting users delete their own history of logins that are older than N days, since it's their data in the end, not mine.

</comment></comment><comment><USER34> My only feedback is to consider pure Debian stable rather than Ubuntu, _particularly_ on the server side. Other than that, it sounds like a pretty decent way to get things done.

<comment><AUTHOR> I'm curious as to the advantages of using Debian over Ubuntu on the server - is it considered generally more stable or better debugged? This is something I've never explored and am interested in your thoughts about it.

At some point in the future, I'd like to give OpenBSD a try.

<comment><USER35> Debian stable moves very slow, which makes it an excellent base OS for a server since predictability dictates that your app brings with it whatever that it needs to run.

For example, nodejs on debian would be outdated, which would force you to setup your app to use a specific version of nodejs that you want to use.

</comment></comment></comment><comment><USER36> So, why not use KeyDB if you need even higher throughput? Supposedly it is supposed to get 5 times the performance.

<comment><AUTHOR> Before your comment I didn't know about the existence of KeyDB, but now I do - thank you for that!

My main issue with scaling Redis is not performance or throughput, but doing it in a controlled, understandable manner which maintains consistency and ideally controlling what information goes on which node. In any case, it's great to find out about KeyDB. Thanks!

<comment><USER36> Thanks for the write up. I didn't know you could use Redis as a primary data store that is reliable enough to put in production. It has really made me curious into how to model outside of a SQL database. My next personal project I think I might try it with KeyDB free tier service (you can host your own, but I would just use the service since it is unlikely to gain steam and is more for having fun).

My mind wants to always normalize the data to the the extreme in a SQL database. So, it will an interesting exercise for myself!

Glad I could introduce you to KeyDB. It is a project I found in the Hacker Newsletter that I've been curious about for a while but I've been thinking SQL, SQL all the time that I never thought I would have a use case for it.

<comment><AUTHOR> I added recently a bit more of detail in the document explaining when I would use redis (and when I wouldn't) to build production systems, here: [https://github.com/fpereiro/backendlore#redis](https://github.com/fpereiro/backendlore#redis) . For most applications, I think it's a viable alternative.

Modeling data with redis is great fun, and quite different to the relational paradigm. I hope you enjoy a lot your upcoming project!

</comment></comment></comment></comment><comment><USER37> Please enlighten me about why Redis is a good use case here.

Won't Redis cache eviction policy make it not a good use case to be a general purpose database, compared to, MongoDB?

<comment><AUTHOR> I added a few more notes on when/why to use Redis and when to use something else: [https://github.com/fpereiro/backendlore#redis](https://github.com/fpereiro/backendlore#redis) . Redis can take you surprisingly far as a general purpose database. I prefer to use either Redis or a relational database, but I'm in no position to tell you that you should do that too, particularly if MongoDB works well for you.

<comment><USER12> Could you elaborate on why your first choice would be Redis? I tend to go for Postgres, but mostly out of habit and familiarity.

<comment><AUTHOR> I love working with fundamental data structures (sets, lists, hashes) much more than working with relational databases. I feel I'm more productive and my code is shorter. It does, however, require far stricter validations in the code; in general a lot of what a relational database does for you (in terms of type validation, schemas and even data consistency), you must do yourself in code or by using redis carefully.

</comment></comment></comment></comment><comment><USER38> I like the openness here. It makes me nervous that the only db used is redis. I prefer a database that's backed by disk files, not one that's backed up occasionally. You get the performance of an in-memory database with redis though.

<comment><USER39> Redis is my favorite database, and it's interesting to me how many people are afraid of it losing data.

It's easy to use both RDB and AOF persistence methods if you want a degree of data safety comparable to what PostgreSQL can provide you. See [https://redis.io/topics/persistence](https://redis.io/topics/persistence)

<comment><USER26> _and it's interesting to me how many people are afraid of it losing data._

Its more about data integrity.

</comment></comment></comment><comment><USER40> A lot of good stuff here I'm gonna steal.

How about leveldb instead of redis to start? One less moving part, though then you might have to migrate to a networked db at some point.

abstract-blob-store might help with the s3/filesystem woes?

<comment><USER40> also, browserify --node or noderify for deployment?

</comment></comment><comment><USER41> Curious, why deal with updating Ubuntu instead of just using heroku?

I've generally found it quite capable of serving nose.js

<comment><AUTHOR> I like dealing with the underlying OS - well, not precisely like, but rather the absence of an intermediate layer between the OS and me. Fortunately, dealing with the OS is very straightforward.

However, if Heroku works well for you, by all means use it! Your time is probably better spent working on your app instead of in the infrastructure. Things that work should only be reconsidered when they start to create real world problems - and the changes that you do in response to real challenges tends to be wiser and more lasting.

</comment></comment><comment><USER9> Nice write-up. However, I would strongly recommend to replace your text-based diagrams, which are difficult to read, with ones produced by relevant tools (e.g., draw.io or a wonderful Mermaid project: [http://mermaid- js.github.io/mermaid](http://mermaid-js.github.io/mermaid)).

<comment><AUTHOR> Thank you for your suggestion! However, I'm a sucker for text diagrams (ever since I saw them in Jonesforth: [https://github.com/nornagon/jonesforth/blob/master/jonesfort...](https://github.com/nornagon/jonesforth/blob/master/jonesforth.S#L199)), so I strongly prefer to leave them as they stand. Hopefully they're still clear enough and get the point across.

<comment><USER33> I was finding your code [1] very compact, reminded me of Forth or those old array languages. This comment kinda explains it :)

[1] [https://github.com/altocodenl/acpic/blob/master/server.js](https://github.com/altocodenl/acpic/blob/master/server.js)

</comment></comment></comment><comment><USER42> Using /tmp for a program's binary files as well as logs seems like a bad idea.

<comment><AUTHOR> Please see [https://github.com/fpereiro/backendlore/issues/5](https://github.com/fpereiro/backendlore/issues/5) .

</comment></comment><comment><USER43> Great design, simple stable and scalable! no need for trendy gimmicks.

</comment><comment><USER44> ...why not use a managed db, eg. AWS RDS from the get-go?

<comment><AUTHOR> It's actually a fine solution and I often do this with infrastructures that I don't intend to maintain myself. In general, however, I try to minimize complexity, and having an external service feels a tad more complex than running the database inside the instance. But you could probably make a successful case to the contrary - namely, that using a service rather than running your own is actually much simpler and easier to maintain.

I must confess that in this decision, the complexity of the AWS console (which I consider awash in buckets of accidental complexity) usually tilts my balance towards self-hosting.

</comment></comment><comment><USER26> Why not start with ELB from the beginning?

<comment><USER45> Because it's easy to add later and you probably don't need it at the beginning.

<comment><USER26> Using ELB(yes, even for one machine) from the start means one less process to worry about, you get HTTPS and don't have to deal with certs, its also cheap.

Since he's striving for "simplicity" and already is in AWS.

<comment><USER17> I feel it is one more thing to worry about as I now have a dependency on something that I can't run locally nor control. It is also more expensive and locked in.

For someone with experience setting up nginx with https from letsencrypt is the same complexity as a hello world. Same thing for running redis yourself instead of adding a cloud provider.</comment></comment></comment></comment>


Nokia faked the still photos too - <AUTHOR> http://sefsar.com/nokia-faked-the-still-photos-too <comment><USER2> [https://sphotos-a.xx.fbcdn.net/hphotos- ash3/578446_101510870...](https://sphotos-a.xx.fbcdn.net/hphotos- ash3/578446_10151087026494473_802634455_n.jpg)

A friend of mine shot this in Helsinki. Low lighting conditions, Nokia style.

<comment><USER3> Thanks. Updated. I'd like to credit him, if he has a preferred link.

<comment><USER2> His request: please add copyright johaee (at) gmail.com. Don't remove the watermark in the image.

</comment></comment></comment><comment><USER4> I can say precisely which lens was used to make that photo: Canon EF 17-40 L USM on a DSLR. Or some other very, very similar lens. No photographer can mistake this for anything.

Like this:

[http://www.flickr.com/photos/meonoctis/5661946485/in/set-721...](http://www.flickr.com/photos/meonoctis/5661946485/in/set-72157628560591547)

<comment><USER5> Wait...why can you say (very, very precisely) that lens (or similar) was the one used? Because of this other picture? I see no resemblance... Would you mind elaborating? I have no idea what you are implying or why.

<comment><USER4> I just mean it is a lens with a very similar optical schema than the Canon 17-40L. The diffraction pattern around light sources is very similar. I played around with this lens some time before with my previous DSLR (now i am a Nikon user) and enjoyed it really. I don't mean it was exactly that lens, but something very close - definitely not fixed f/2.0 cheap lens of a smartphone, that kind of halo with 'rays' around light sources is possible only with a lens having many internal components which will be too thick to fit into a smartphone, and not f/2.0, and of course too expensive for a smartphone.

So i fully agree with the article saying it was a fake.

<comment><USER5> Gotcha. Definitely didn't doubt the article; just wondering what you meant. Thanks!

<comment><USER4> Also the article is most likely wrong about calling these artifacts purely diffraction-related, creating diffraction images that big and visible would require too small focal ratio, which is unlikely for night conditions, because they increase exposure time which is already too long. These are just a consequence of internal back-reflections inside the lens optics which manifest itself on extreme contrast parts of image (bright light vs dark sky). Especially obvious on astronomical images, where control of focal ratio is manual and it's almost always open fully to collect as much light as possible - of course, focus depth is of no meaning for astronomy - and the quality of this lens allows for perfect images even when wide open. Bright stars always looks like this on Canon 14-40L.

<comment><USER6> i don't understand what you are saying here. astronomical images don't look exactly like that - they have fewer "spikes" - but they are similar. the spikes are either from diffraction caused by the secondary mirror support (which light passes through in a reflecting telescope) or readout artifacts in the CCD, particularly when it saturates (although that looks quite different - a single vertical line, often).

i agree that the spikes (in astronomy images) don't come from blade edges in a diaphragm aperture, but they are still diffraction spikes (from the secondary support) and "everyone" (as far as i know / remember) refers to them as such.

you can see the secondary support in the top right image at <http://en.wikipedia.org/wiki/Reflecting_telescope> \- the four radial "things" that hold the cylinder in the "end" of the telescope (on the right).

so what are you referring to with "internal back reflections" and how does that give spikes (rather than ghosts)?

[above from (past) personal experience as an optical astronomer on research telescopes - i am not sure if you are confused, or if there's some kind of problem i never understood. perhaps you work on a system where this is an issue?]

<comment><USER4> I am speaking about astronomical images not in general, but those taken with the Canon EF 17-40 f/4 L USM lens. They are almost always taken with the diaphragm wide open, and contain same kind of spikes -> spikes are not from diffraction.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> Faking images for a product rollout appears to be common mobile-electronics industry practice. Apple faked a video still from _Star Trek_ and made a fake video of the _New York Times_ for their iPad introduction.

[http://news.softpedia.com/news/Apple-Exposed-As-Liars-in- Fak...](http://news.softpedia.com/news/Apple-Exposed-As-Liars-in-Faked-iPad- Promo-146566.shtml)

[http://www.macrumors.com/2010/02/22/apples-ipad-nytimes- prom...](http://www.macrumors.com/2010/02/22/apples-ipad-nytimes-promo-video- shows-flash-again/)

I wonder if Nokia cross-licensed Apple's patent.

<comment><USER8> Id argue that there is a difference of intent in these two cases. Nokia is deceiving customers about a specific function or capability (camera) while Apple is just showing a promotional video of the device as a whole. Both cases are reprehensible, no doubt, but maybe I'm the only one who think that Nokia has gone a little too far.

Lets think of these in terms of food. Its not the same to show a "perfect Big Mac" in and ad than it is to show a Big Mac thats made with completely different ingredients of the ones you'll find when you order one.

<comment><USER7> Apple showed the iPad displaying Flash content as a way of showcasing the browsing experience consumers could expect. In addition they manipulated a still image to sell the iPad as a device for watching movies. The choice of a still image was based upon an intent to deceive potential customers about the shortcomings of the 4:3 format of the iPad.

In the case of the iPad watching feature films is probably about equivalent to using the camera on a smartphone. On the other hand, browsing the web is a primary function of the device, and there was no way the iPad was getting Flash.

</comment></comment></comment><comment><USER9> "Professional driver on closed course."

"Screen image is simulated."

"Product enlarged to show detail."

etc. etc.

Yes, yes, we like "truth" and all. Especially when they're selling a camera after all, but getting a great cameraphone image and getting an image that will work on tv are two different things. No real surprise. Maybe disappointment, but no suprise.

<comment><USER10> Except there's no implying or warning, just outright lying about this being a sample image.

<comment><USER9> Agreed. They should have small disclaimer text at the bottom. And that's what everyone should be up in arms about, not that they faked it in the first place.

I know there are rules for food advertising, but don't know about other types of products... There's not a safety liability issue like with cars, but.

<comment><USER11> Also these images are _nothing_ like the images the camera would take. That is like advertising a totally different car than the one you are selling.

<comment><USER12> Exactly. It's like getting cockpit footage from a nascar and using it in a ford focus commercial.

</comment></comment></comment></comment></comment><comment><USER13> _How to botch your annual device announcement; you know the one the whole company is riding on_ by Stephen Elop with a special Forward by the guys who ran RIM into the shitter.

</comment><comment><USER14> Here is an apology from Nokia: <http://conversations.nokia.com/2012/09/06/an- apology-is-due/>

Here are videos (OIS off and on) shot with a Lumia 920: [http://www.youtube.com/watch?feature=player_embedded&v=6...](http://www.youtube.com/watch?feature=player_embedded&v=6HSbhyaH0vw)

<comment><USER15> Even the apology for being vague and somewhat underhanded is somewhat.. underhanded.

      Here is the video shown at the press conference shot using a Lumia 920. On the right is a Lumia 920 prototype with OIS. On the left is a smartphone without OIS. The difference is apparent.

Notice they specify 'a smartphone'. The wording shifts the focus from the camera to OIS, and then they must have used a completely different phone for the comparison. Weird.

<comment><USER16> It could have been an older Nokia Lumia, but they wouldn't want to admit that publicly.

</comment></comment></comment><comment><USER17> In case the HN submission of the following doesn't float to the front page, it seems Nokia has apologized for the fake advertisement:

<http://www.bbc.com/news/technology-19499879>

EDIT: The HN submission

<https://news.ycombinator.com/item?id=4483817>

<comment><USER18> They're not sorry for the lies and intentional deceit. They're just sorry they got caught.

</comment></comment><comment><USER19> There's only one way out of this for Nokia. They need to release new photo's and video's created with the new Nokia Wndows 8 Phone right the f __* now. With as much EXIF and/or metadata as possible. If I was Nokia I'd hire the best photographer-artist the world knows and make a big splash about it.

<comment><USER15> Or they could just pretend this never happened.

A search on Google News (anecdotal, yes) [https://www.google.com/search?hl=en&gl=us&tbm=nws...](https://www.google.com/search?hl=en&gl=us&tbm=nws&q=nokia+lumia+920&oq=nokia&gs_l=news- cc.3.1.43j0i3j0l8j43i400.1672.2343.0.4522.214.171.124.188.8.131.52.288.3j1.4.0...0.0...1ac.RvsI4Bj0wHg)

None of the articles have anything related to this in the title, and the first three articles do not mention any faking.

It feels like a big issue in tech-savvy circles on the internet. But I doubt most consumers know or care.

<comment><USER20> The second result for me (posted 7 minutes ago) is related to this topic: <http://i.imgur.com/oJQYf.png>

</comment></comment></comment><comment><USER21> Funny that people ignore the fake facetime video's of iphone because it's from apple, and keep bashing microsoft because it's microsoft.

Fake advertisements, it's nothing new, EVERYBODY does that. Think real

<comment><USER22> Apple has a disclaimer about "simulated" images in most of their commercials, and if you look at their dedicated facetime ads [1] where they don't have a disclaimer, you'll notice...that is how facetime actually looks and works.

[1] <http://youtu.be/RxVzP_OWnDw?hd=1>

<comment><USER23> *Sequences shortened

</comment></comment></comment><comment><USER24> There is definitely something up. The Register got their hands on one but the minders managed to stop them sending the photo externally:

[http://www.theregister.co.uk/2012/09/05/handson_nokia_lumia_...](http://www.theregister.co.uk/2012/09/05/handson_nokia_lumia_920_windows_phone_8/)

 _"Alas, the Minders leapt into Red Alert mode to prevent me transmitting the results, you'll have to take my word for it."_

<comment><USER25> Andrew Orlowski has to be one of the most unreliable troll journalists and last person who's word I will take at face value.

<comment><USER24> I agree and I take things he says lightly, but he's right on this one.

</comment></comment></comment><comment><USER26> I don't understand why they need this.

They made a gorgeous and great device. It probably has one of the best smartphone cameras ever. Because of what they did, now everyone is talking about how they faked the photos instead of talking about the phone. A friend of mine even emailed Nokia and asked "Will the Lumia 920 also have the built- in CIV (cameraman in van) feature?"

Apologizing isn't going to change anything. People are going to keep making fun of Nokia for a long time about this.

</comment><comment><USER27> Nokia has been very very misleading about pretty much everything their phone can do, from the PureView naming to how the actual photos and videos look like.

</comment><comment><USER28> What's sad is not only the dishonesty about the material, but what it says about the product.

Do they simply not have enough confidence in their own flagship smartphone, with an apparently world-beating camera, to show an actual unretouched shot? A great way to crap all over the launch.

<comment><USER16> I think this is more of just a pigheaded move by the advertising team.

Nokia camera's have been good in the past and I have no doubt this is probably a good one.

One ad campaign is short lived, the photos taken by consumers who buy the phone on the other hand will continue to trickle in. It doesn't make too much sense for them to rely on this fact.

What the whole brouhaha is that they probably thought 'It's just an ad, everyone does it, every knows how ice cream in ice cream ads is not real ice cream. And the delicious burger never looks the same in real life either.'

Here's an interesting video from McDonalds on how they make their burgers look great. They admit the shortcomings of a real burger in the looks department and how they make it look so good.

<http://youtu.be/oSd0keSj2W8>

<comment><USER29> In most markets the ice cream in an ice cream ad has to be ice cream. The ice cream in non-ice cream ads, on the other hand, can be anything.

<comment><USER30> actually, i read some where regarding tips in food photography.. where they dont actually use ice cream.. let me find you the link.. it is more of an advertising thing..

<comment><USER31> The gist is that the item being advertised has to be the actual item as offered to the consumer. So in a cereal ad, the cereal featured has to be the actual cereal (even though they may go through hundreds of boxes picking out the perfect corn flakes to use). However, since they're not actually selling the milk that goes with the cereal, they're free to use glue or some other liquid that will let the cereal maintain it's crunch or otherwise enhance photography.

<comment><USER32> So they are selling real flakes and a fraudulent crunch. Still fraud.

</comment></comment></comment></comment></comment></comment><comment><USER33> This is too funny. And on top of all this, the sheer comedy of the ad. I mean whats going on here? Is this supposed to be some date night? A couple on vacation? No. Felt more like a guy who somehow landed a girlfriend/date above his league and is trying to get as much as possible on camera to show his friends because they won't believe him.

<comment><USER34> That, and the example shots all look like someone picked the wrong LASIK surgeon.

</comment></comment><comment><USER35> No way... Fake pictures in ads? Who would have thought!

<comment><USER36> I've heard[0] that 'big' camera makes like Canon or Nikon regularly advertise their products using photos made with completely different equipment. It would be more surprising if Nokia used actual photos made with Lumia in promotion materials, I guess.

[0] <http://www.kenrockwell.com/nikon/d3/h3d-1.htm>: "For those of us in the press (Ken), we laughed when we looked at the EXIF data of Nikon's PR photos of the D3. They were shot in a studio, so of course Nikon had them shot with an appropriate camera: a $30,000 Phase One P45 back." Anecdotal evidence, but I'm unable to find another source, so take it with a grain of salt.

<comment><USER37> Rockwell is a noted self-publicist with a politician's reputation for truth and accuracy. Nikon can take their PR shots with whatever is best, should P&S publicity shots be taken with that P&S model? Of course not. If on the other they take photos with professional equipment and claim or imply that it was taken with a mobile phone then that is deceitful.

<comment><USER36> Yes, I probably had confused shots _of_ the camera vs. shots _made with_ the camera. This was ignorant of me.

Though will we consider a deceit if the camera used is the same P&S model, but lighting is set up? And then there's extensive post-processing, which is possible and may alter shots significantly.

</comment></comment></comment></comment><comment> Ono-Sendai A small camera with a small aperture and smaller sensor than a DSLR will produce a greater amount of aperture diffraction than a DSLR, at the same f-value.

</comment><comment><USER38> [http://yle.fi/uutiset/nokia_apologises_for_misleading_lumia_...](http://yle.fi/uutiset/nokia_apologises_for_misleading_lumia_video/6284588?origin=rss) Confusion? This is deceit! A video of just OIS? The Video states at the end, "This is Lumia". It seems the once honest Nokia has picked up some bad habits from their partnership with MicroSoft.

</comment><comment><USER39> Why so much hate on Nokia?

<comment><USER40> Because clearly, they lie about what their products can do.

<comment><USER41> People have done that since the dawn of radio adverts and billboards...

<comment><USER42> Which doesn't mean it shouldn't be penalized.

</comment></comment></comment></comment><comment><USER43> When I have a look on the photo, I think that the depth has to be remarkably long. That is because the sign "RESTAURANT" looks very clear. In other words small aperture and short exposure time. This is not taken by Lumia camera shot.

</comment><comment><USER43> Kirjoitin jo englanniksi. Katuvalojen lisäksi kiinnitän huomiota kylttiin "RAVINTOLA". Pitkällä valotusajalla se "Neonvalo" tms puuroutuisi. Nyt se näkyy selkeästi eli kyseessä on pieni aukko ja lyhyt valotusaika.

</comment><comment><USER44> Nokia N97 promo video was also riddled with fakeness <http://news.ycombinator.com/item?id=1231464>

</comment><comment><USER45> So the final photo in that article _is_ actually shot with the real Nokia phone camera?

Because it looks really good. They didn't need to have faked anything.

</comment><comment><USER46> I wonder why did they do that, it was pretty obvious. The second picture looks good enough.

</comment><comment><USER47> I might be able to take that photo on that phone. I would use a high-output continuous light balanced with gels to match the ambient colour, and create diffraction stars by putting a cross filter in front of the lens. I think it's fake too, but I'd have a go at making it.</comment>


PayPal and Two-Factor Authentication: A "Weakest Link" Case in Point - <AUTHOR> http://danielmiessler.com/blog/paypal-and-two-factor-authentication-a-weakest-link-case-in-point <comment><USER2> ... and no matter what you do, your username + password will always work using the PayPal API without the second factor auth. This is true also of E*TRADE and Bank of America.

<comment><AUTHOR> Man, that's weak. I'm all for single-factor, if that's all that's available, but don't sell two-factor and implement it so it's an illusion.

</comment></comment><comment><USER3> The primary problem is that everyone is stuck on this Username+Password mentality. This is an ancient model that needs a rethink.

I'm working on a solution to replace web site logins. Please ping me if you're interested or have some expertise in this area. Shamenski of Gmail.

</comment><comment><USER4> I get a 404. Try [http://danielmiessler.com/blog/verisign-vip-a-weakest- link-c...](http://danielmiessler.com/blog/verisign-vip-a-weakest-link-case-in- point)</comment>


Show HN: Dude PA – Personal assistant based on the command line - <AUTHOR> https://github.com/dude-pa/dude <comment><USER2> If it meets the guidelines, this might make a good 'Show HN'. Show HN guidelines: [https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html)

<comment><AUTHOR> Hi, I don't understand. This is a 'Show HN'

<comment><USER2> Sometimes, I am an idiot.

</comment></comment></comment><comment><AUTHOR> Hi all, This is a side project I have been working on since a few months. Since I am fairly new to production level programming, I would be very thankful if you could critique the project by giving advice related to coding practices, feature suggestions, or anything else you think is necessary.

</comment><comment><USER3> I might be willing to take a closer look at it, if this wasn't named "dude".

Dude, do you even?

<comment><AUTHOR> Haha, I'll keep this in mind the next time I implement an idea.</comment></comment>


Ask HN:  Novels about or featuring malevolent AI? - <AUTHOR>

I am looking to read some good hard SF about malevolent AI.  Any suggestions? <comment><USER2> 1- Gregory Benford's "Galactic Center" series <http://en.wikipedia.org/wiki/Galactic_Center_Saga>

2- Dan Simmons' "Hyperion Cantos" series <http://en.wikipedia.org/wiki/Hyperion_Cantos>

</comment><comment><USER3> Bungie's Marathon trilogy, although not a novel, explored the inevitable digression of AI into rampancy. The terminal transcripts can still be found over here: <http://marathon.bungie.org/story/contents.html>

</comment><comment><USER4> _Robopocalypse_ , Daniel Wilson.

[http://www.amazon.com/Robopocalypse-Novel-Daniel-H- Wilson/dp...](http://www.amazon.com/Robopocalypse-Novel-Daniel-H- Wilson/dp/0385533853)

</comment><comment><USER5> The AIs in Vernor Vinge's A Fire Upon the Deep can be pretty nasty.

</comment><comment><USER6> Not sure about hard SF, but Prey by Michael Crichton?

<http://en.wikipedia.org/wiki/Prey_(novel)>

</comment><comment> benji-york How about one where the AI is inadvertently malevolent? <http://localroger.com/prime-intellect/index.html>

</comment><comment><USER7> I have no mouth and I must scream, by Harlan Ellison.</comment>


Ask HN: Do I need to worry about bit rot in 2015? - <AUTHOR>

Hi HN,
Under what situations should I worry about bit rot in 2015 rather than just outright disk failure?
For my photo album? For my postgres database with important data?
Does SSD make a difference? <comment><USER2> Not sure if you're already aware, but "bit rot" is an ambiguous term nowadays.

The original definition[1] refers jokingly to bits "rotting" as an (obviously false if taken literally) explanation of why software that used to work fine has stopped working after a period of time elapsed, actually because of causes now more properly(?) known as "software rot"[2], but the term is sometimes (mistakenly IMHO) used to refer to actual data degradation[3], or "data rot"[3]

This whole sorry tale is actually due to the related but little-known phenomenon of "sememe rot", a gradual decay in units of meaning so that thoughts and sentences that used to be perfectly coherent will gradually become incomprehensible, or even worse come to mean something different, even quite opposite, to their original import.

Anyway, from your SSD question I guess you're interested in data degradation. The best advice here is just the obvious: "always back up!" Though this doesn't answer the interesting question as to whether data degradation is more or less of a problem this year than last, it is still the best strategy.

[1] [http://www.catb.org/jargon/html/B/bit- rot.html](http://www.catb.org/jargon/html/B/bit-rot.html)

[2] [http://en.wikipedia.org/wiki/Software_rot](http://en.wikipedia.org/wiki/Software_rot)

[3] [http://en.wikipedia.org/wiki/Data_degradation](http://en.wikipedia.org/wiki/Data_degradation)</comment>