Ask HN: How was your experience with AWS Lambda in production? - <AUTHOR> I would like to hear from people who have used AWS Lambda in production, how was there experience with it. It would be great if you have references to project repositories. <comment><USER2> I made an image hosting tool on Lambda and S3, for internal corporate use. Staff can upload images to S3 via an SPA. The front end contacts the Lambda service to request a pre-signed S3 upload URL, so the browser can upload directly to S3. It works really well. Observations:

1\. Took too long to get something working. The common use case of hooking up a Lambda function to an HTTP endpoint is surprisingly fiddly and manual.

2\. Very painful logging/monitoring.

3\. The Node.js version of Lambda has a weird and ugly API that feels like it was designed by a comittee with little knowledge of Node.js idioms.

4\. The Serverless framework produces a huge bundle unless you spend a lot of effort optimising it. It's also very slow to deploy incremental changes _edit:_ – this is not only due to the large bundle size but also due to having to re-up the whole generated CloudFormation stack for most updates.

5\. It was worth it in the end for making a useful little service that will exist forever with ultra-low running costs, but the developer experience could have been miles better, and I wouldn't want to have to work on that codebase again.

\---

Edit: here's the code: [https://github.com/Financial-Times/ig-images- backend](https://github.com/Financial-Times/ig-images-backend)

To address point 3 above, I wrote a wrapper function (in src/index.js) so I could write each HTTP Lambda endpoint as a straight async function that simply receives a single argument (the request event) and asynchronously _returns_ the complete HTTP response. This wouldn't be good if you were returning a large response though; you'd probably be better streaming it.

<comment><USER3> The problem of logs is actually a problem of CloudWatch Logs being just not a very good service. A great way to solve that is to push all logs from CloudWatch Logs into an ElasticSearch cluster (using a Lambda function). AWS even has the code already done for you if you click the "subscribe" button in CWL. Then with Kibana/ElasticSearch the experience of inspecting and analysing logs is MUCH better.

<comment><USER4> We built IOpipe[1] to address these issues by offering our own wrapper[2] that sends telemetry to our service. IOpipe aggregates metrics, and errors, and allows the creation of alerts with multiple rules per alert.

[1] - [https://iopipe.com](https://iopipe.com) [2] - [https://github.com/iopipe/iopipe/](https://github.com/iopipe/iopipe/)

<comment><USER5> +1 to the folks at IOpipe. A really cool product that gives you very interesting visibility into your Lambda function executions!

</comment></comment></comment></comment><comment><USER6> We use AWS Lambda to process Hearthstone replay files.

My #1 concern with it went away a while back when Amazon _finally_ added support for Python 3 (3.6).

It behaved as advertised: Allowed us to scale without worrying about scaling. After a year of using it however I'm really not a big fan of the technology.

It's opaque. Pulling logs, crashes and metrics out of it is like pulling teeth. There's a _lot_ of bells and whistles which are just missing. And the weirdest thing to me is how people keep using it to create "serverless websites" when that is really not its strength -- its strength is in distributed processing; in other words, long-running CPU-bound apps.

The dev experience is poor. We had to build our own system to deploy our builds to Lambda. Build our own canary/rollback system, etc. With Zappa it's better nowadays although for the longest time it didn't really support non- website-like Lambda apps.

It's _expensive_. You pay for invocations, you pay for running speed, and all of this is super hard to read on the bill (which function costs me the most and when? Gotta do your own advanced bill graphing for that). And if you want more CPU, you have to also increase memory; so right now our apps are paying for hundreds of MBs of memory we're not using just because it makes sense to pay for the extra CPU. (2x your CPU to 2x your speed is a net-neutral cost, if you're CPU-bound).

But the kicker in all this is that the entire system is proprietary and it's really hard to reproduce a test environment for it. The LambCI people have done it, but even so, it's a hell of a system to mock and has a pretty strong lock-in.

We're currently moving some S3-bound queue stuff into SQS and dropping Lambda at the same time could make sense.

I certainly recommend trying Lambda as a tech project, but I would not recommend going out of your way to use it just so you can be "serverless". Consider your use case carefully.

<comment><USER7> Can you clarify this point:

> its strength is in distributed processing; in other words, long-running CPU- > bound apps.

it seems to me that's an explicit non-usecase for Lambda given it limits sessions to < 5min per invocation.

<comment><USER6> I should have been more specific yes. I actually meant CPU-or-network-bound tasks with a predictable, sub-5min runtime.

Like I said we use it for game replay processing, so that's 5-15 second tasks that read and parse log files and hit a db and s3 with the results.

Other suitable tasks: image resizing, API chatter, bounded video transcoding, etc. Lambda is pretty good at distributed processing (as long as you can make the bill work in your favour, over hosting your own overprovisioned fleet).

</comment></comment></comment><comment><USER5> Hey all, My name is Chris Munns and I am currently the lead Developer Advocate for Serverless at AWS (I am part of the Lambda PM team). We really appreciate this feedback and are always looking for ways to hear about these pain points. Can email me directly: firstname.lastname@example.com if you ever get stuck.

Thanks, \- Chris

<comment><USER8> I am on a devops team trying to implement lambda by splitting pieces off from a monolithic Java app where applicable.

The biggest pain point I have is because we have multiple "environments" (such as dev, da, staging) in the same amazon account and because lambdas are global I can't limit access to resources via IAM easily without hacks.

Aka, because the same lambda will be used (but different versions and/or aliases) on all environments I can't marry the code and configuration to limit access to say RDS or elasticache or an S3 bucket per lambda.

I feel like I need a higher order primitive (a lambda group that is role + configuration can live in that includes the lambdas) to achieve this. I realize api gateway has the concepts of stages but currently the idea is for some lambdas to be invoked directly by the monolithic app or via SNS/SQS async.

Otherwise I could namespace my lambda functions which is hacky and make DevFooBar, StageFooBar, etc.

Currently we plan to split off our environments into separate AWS accounts.

<comment><USER5> Hi Runamok,

Today given that aliases/versions do not have any sort of different permissions you are probably best to either run completely different stacks of resources or the multiple account model. With AWS Organizations these days its not that hard to run multiple environments across accounts.

We did a webinar on some of this a few months back, the slides here might be useful to you: [https://www.slideshare.net/AmazonWebServices/building-a- deve...](https://www.slideshare.net/AmazonWebServices/building-a-development- workflow-for-serverless-applications-march-2017-aws-online-tech-talks)

It heavily leverages AWS's tools, but you could create similar practices using 3rd party frameworks and CI/CD tools as well.

Thanks,

-munns

</comment></comment></comment><comment><USER9> We use Lambda for 100% of our APIs some of which get over 100,000 calls per day. The system is fantastic for micro services and web apps. One caveat, you must use a framework like Serverless or Zappa. Simply setting up API Gateway right is a hideous task and giving your function the right access level isn't any fun either. Since the frameworks do all that for you it really makes life easier.

One thing to note. API Gateway is super picky about your response. When you first get started you may have a Lambda that runs your test just fine but fails on deployment. Make sure you troubleshoot your response rather than diving into your code.

I saw some people complaining about using an archaic version of Node. This is no longer true. Lambdas support Node V6 which, while not bang up to date, is an excellent version.

Anyway, I can attest it is production ready and at least in our usage an order of magnitude cheaper.

</comment><comment><USER10> I worked on a project where the architect wanted to use Lambdas for the entire solution. This was a bad choice.

Lambdas have a lot of benefits - for occasional tasks they are essentially free, the simple programming model makes them easy to understand in teams, you get Amazon's scaling and there's decent integration with caching and logging.

However, especially since I had to use them for whole solution, I ran into a ton of limitations. Since they are so simple, you have to pull in a lot of dependencies which negate a lot of the ease of understanding I mentioned before. The dependencies are things like Amazon's API Gateway, AWS Step Functions, and AWS CLI itself, which is pretty low-level. So now, the application logic is pretty easy, but now you are dealing with a lot of integration devops. There's API Gateway is pretty clunky and surprisingly slow. Lambdas shut themselves down, and restarting is slow. The Step Functions have a relatively small payload limit that needs to be worked around. Etc. So use them sparingly!

<comment><USER11> 1\. Use a framework for deploying your lambda functions. There are a few that will manage the API Gateway for you.

2\. Don't put the Lambda inside a VPC if you want lower response times

3\. Step Functions don't seem ready for prime time that I can tell. (This might have changed in the last couple of months)

4\. Lambda Functions should be microservices. Small and lean.

5\. There is a limit on resources for CloudFormation so at about 20-30 functions with API Gateway on the serverless framework, you will hit a limit and can't add anymore (other deployment tools which don't use CloudFormation shouldn't have an issue)

6\. Want more CPU, add more RAM

<comment><USER12> What makes you say Step Functions isn't ready for prime time? We've been using it (and SWF which it is based off) for about a year at decent scale and been generally very happy with it.

<comment><USER11> When I initially set the up, there was no way to edit or delete them. I haven't messed with them since, although, I did just check and there aren't any step functions in my account so it looks like they deleted the old ones.

It may be time to test them out again, I just go bit pretty bad with the last time I implemented them and lost about a weeks worth of work because of it being un-usable.

</comment></comment></comment></comment><comment><USER13> \- Monitoring & debugging is little hard

\- CPU power also scales with Memory, you might need to increase it to get better responses

\- Ability to attach many streams (Kinesis, Dynamo) is very helpful, and it scales easily without explicitly managing servers

\- There can be a overhead, your function gets paused (if no data incoming) or can be killed undeterministically (even if it works all the time or per hour) and causes cold start, and cold start is very bad for Java

\- You need to make your JARs smaller (50MB), you cannot just embed anything you like without careful consideration

<comment><USER14> @CSDude you can check out this small tool i've written for debugging Lambda and API Gateway integration.

[https://github.com/AlexanderC/lambdon](https://github.com/AlexanderC/lambdon) (i know the name sucks)

Also @chetanmelkani as a hint: if you are using NodeJS runtime most optimal from the execution time and cost efficiency perspective is setting up 512mb of memory ;) it's about getting x2 performance boost over the 128mb configuration.

</comment></comment><comment><USER15> I deployed a couple AWS lambda endpoints for very low-volume tasks using claudia.js - Claudia _greatly_ reduces the setup overhead for sane REST endpoints. It creates the correct IAM permissions, gateway API and mappings.

Claudia.js also has an API layer that makes it look very similar to express.js versus the weird API that Amazon provides. I would not use lambda + JS without claudia.

For usage scenarios, one endpoint is used for a "contact us" form on a static website, another we use to transform requests to fetch and store artifacts on S3. I can't speak toward latency or high volume but since I've set them up I've been able to pretty much forget about them and they work as intended.

<comment><USER16> Oh claudia sounds interesting. I'm a mobile developer who's used express/node before. The hardest thing for me when I was cobbling a backend together with lambda/dynamodb was unterstanding the permission system and debugging it when I configured it wrong.

Lots of the examples and articles around this process are out of date and AWS's web front end can be painful to deal with. That said, when everything was setup, it was pretty straight forward to maintain.

Do you have any links to good tutorials on Claudia? I'd love to setup a contact me form using lambda for a project I'm working on.

Also, do you know how dos Claudia compare to stuff like serverless.js?

<comment><USER17> Hey, one of the guys from Claudia here.

You can have tutorials and examples on the : \- Claudia.js website - [https://claudiajs.com](https://claudiajs.com) \- Claudia Github examples - [https://github.com/claudiajs/example- projects](https://github.com/claudiajs/example-projects)

The purpose of Claudia.js is just to make it super easy to develop and deploy your applications on AWS Lambdas, API Gateway, also ease up the work with DynamoDb, AWS IoT, Alexa and so on.

There are two additional libraries: Claudia API Builder and Claudia Bot Builder, to ease up API and chat bot development and deployment.

Regarding the contract form - the best is to create a single service that will handle all the contract form requests. At that point, you can either connect it to DynamoDb, or even call some other data storage / service.

Both Serverless and Claudia have their points where they shine. For a better understanding of their comparison, you can read about it in the Claudia FAQ - [https://github.com/claudiajs/claudia/blob/master/FAQ.md#how-...](https://github.com/claudiajs/claudia/blob/master/FAQ.md#how- does-it-compare-to-)

<comment><USER16> Awesome answer, thanks!

</comment></comment></comment></comment><comment><USER18> We have a number of different use cases at FundApps, some obvious like automated tasks, automatic DNS, cleanup AMI's etc, to the more focused importing and parsing of data from data sources. This is generally a several times a day operation, so lambda was the right choice for us. We also use API gateway with lambdas, its a small API, about 2 requests per second on average, but very peaky during business hours, its response and uptime has been excellent.

Development can be tricky, there are a lot of of all in one solutions like the serverless framework, we use Apex CLI tool for deploying and Terraform for infra. These tools offer a nice workflow for most developers.

Logging is annoying, its all cloudwatch, but we use a lambda to send all our cloudwatch logs to sumologic. We use cloudwatch for metrics, however we have a grafana dashboard for actually looking at those metrics. For exceptions we use Sentry.

Resources have bitten us the most, not enough memory suddenly because the payload from a download. I wish lambda allowed for scaling on a second attempt so that you could bump its resources, this is something to consider carefully.

Encryption of environment variables is still not a solved issue, if everyone has access to the AWS console, everyone can view your env vars, so if you want to store a DB password somewhere, it will have to be KMS, which is not a bad thing, this is usually pretty quick, but does add overhead to the execution time.

</comment><comment><USER19> I'm running Rust on Lambda at the moment for a PBE board gaming service I run. I can't say it runs at huge scale though, but using Lambda has provided me with some really good architectural benefits:

* Games are developed as command line tools which use JSON for input and output. They're pure so the game state is passed in as part of the request. An example is my implementation of Lost Cities[1]

* Games are automatically bundled up with a NodeJS runner[2] and deployed to Lambda using Travis CI[3]

* I use API Gateway to point to the Lambda function, one endpoint per game, and I version the endpoints if the game data structures ever change.

* I have a central API server[4] which I run on Elastic Beanstalk and RDS. Games are registered inside the database and whenever players make plays, Lambda functions are called to process the play.

I'm also planning to run bots as Lambda functions similar to how games are implemented, but am yet to get it fully operational.

Apart from stumbling a lot setting it up, I'm really happy with how it's all working together. If I ever get more traction I'll be interesting to see how it scales up.

[1]: [https://github.com/brdgme/lost-cities](https://github.com/brdgme/lost- cities)

[2]: [https://github.com/brdgme/lost- cities/blob/master/.travis.ym...](https://github.com/brdgme/lost- cities/blob/master/.travis.yml)

[3]: [https://github.com/brdgme/lambda/blob/master/index.js](https://github.com/brdgme/lambda/blob/master/index.js)

[4]: [https://github.com/brdgme/api](https://github.com/brdgme/api)

<comment><USER20> I'm not sure if you care, but it's possible to use Neon to write native Node modules in Rust. No need for JS glue code and better performance than spawning a process.

<comment><USER19> Will check that out, cheers!

</comment></comment></comment><comment><USER21> It gets the job done but the developer experience around it is awful.

Terrible deploy process, especially if your package is over 50mb (then you need to get S3 involved). Debugging and local testing is a nightmare. Cloudwatch Logs aren't that bad (you can easily search for terms).

We have been using Lambdas in production for about a year and a half now, to do 5 or so tasks. Ranging from indexing items in Elasticseaech, to small CRON clean up jobs.

One big gripe around Lambads and integration with API Gateway is they totally changed the way it works. It use to be really simple to hook up a lambda to a public facing URL so you could trigger it with a REST call. Now you have to do this extra dance with configuring API Gateway per HTTP resource, therefore complicating the Lambda code side of things. Sure with more customization you have more complexity associated with it, but the barrier to entry was significantly increased.

</comment><comment><USER22> I've been using AWS Lambda on a side project (octodocs.com) that is powered by Django and uses Zappa to manage deployments.

I was initially attracted to it as a low-cost tool to run a database (RDS) powered service side project.

Some thoughts:

\- Zappa is a great tool. They added async task support [1] which replaced the need for celery or rq. Setting up https with let's encrypt takes less than 15 minutes. They added Python 3 support quickly after it was announced. Setting up a test environment is pretty trivial. I set up a separate staging site which helps to debug a bunch of the orchestration settings. I also built a small CLI [2] to help set environment variables (heroku-esque) via S3 which works well. Overall, the tooling feels solid. I can't imagine using raw Lambda without a tool like Zappa.

\- While Lambda itself is not too expensive, AWS can sneak in some additional costs. For example, allowing Lambda to reach out to other services in the VPC (RDS) or to the Internet, requires a bunch of route tables, subnets and a nat gateway. For this side project, this currently costs way more running and invoking Lambda.

\- Debugging can be a pain. Things like Sentry [3] make it better for runtime issues, but orchestration issues are still very trail and error.

\- There can be overhead if your function goes "cold" (i.e. infrequent usage). Zappa lets you keep sites warm (additional cost), but a cold start adds a couple of seconds to the first-page load for that user. This applies more to low volume traffic sites.

Overall: It's definitely overkilled for a side project like this, but I could see the economics of scale kicking in for multiple or high volume apps.

[1]: [https://blog.zappa.io/posts/zappa-introduces-seamless- asynch...](https://blog.zappa.io/posts/zappa-introduces-seamless-asynchronous- task-execution)

[2]: [https://github.com/cameronmaske/s3env](https://github.com/cameronmaske/s3env)

[3]: [https://getsentry.com/](https://getsentry.com/)

<comment><USER23> Zappa author here, thank you for your kind recommendation!

Lots more features in the pipeline, too!

<comment><USER24> I usually default to using Flask for Python when building APIs and using Zappa to deploy it has been a wonderful experience. Easy to develop locally with a Flask web server and then you just deploy with Zappa.

I haven't used it in a huge production environment, but it's definitely my go to way of handling APIs in side projects and other related things.

</comment></comment></comment><comment><USER25> I've been using it for heavy background job for [http://thefeed.press](http://thefeed.press) and overall, I think it's pretty ok (I use NodeJs). That said here are few things:

\- No straight way to prevent retries. (Retries can crazily increase your bill if something goes wrong)

\- API gateway to Lambda can be better. (For one, Multipart form-data support for API gateway is a mess)

\- (For NodeJs) I don't see why the node_modules folder should be uploaded. (Google cloud functions downloads the modules from the package.json)

<comment><USER26> > I don't see why the node_modules folder should be uploaded.

Exactly! Especially if you're using modules that include some sort of binary and build your function on macOS it's a pain -- I ended up using a Docker- based workflow to get the correct binaries into the node_modules.

<comment><USER27> Or you can use CI to deploy.

</comment></comment></comment><comment><USER28> I'd recommend using a framework such as the Serverless Framework[1], Chalice[2], Dawson[3], or Zappa[4]. As any other (web) development project, using a framework will alleviate a big part of the pain involved with a new technology.

Anyways, I'd recommend starting from learning the tools without using a framework first. You can find two coding sessions I published on Youtube[5][6].

[1]: [https://serverless.com/](https://serverless.com/)

[2]: [https://github.com/awslabs/chalice](https://github.com/awslabs/chalice)

[3]: [https://dawson.sh/](https://dawson.sh/)

[4]: [https://github.com/Miserlou/Zappa](https://github.com/Miserlou/Zappa)

[5]: [https://www.youtube.com/watch?v=NhGEik26324](https://www.youtube.com/watch?v=NhGEik26324)

[6]: [https://www.youtube.com/watch?v=NlZjTn9SaWg](https://www.youtube.com/watch?v=NlZjTn9SaWg)

</comment><comment><USER29> At Annsec we are all out on serverless infrastructure and use Lambdas and Step functions in two development teams on a single backlog. Extensibility of a well written lambda is fenomenal. For instance we have higher abstraction lambdas for moving data. We make them handle several input events and to the greatest extent as pure as possible. Composing these lambdas later in Step functions is true developer joy. We unit test them locally and for E2E-tests we have a full clone of our environment. In total we build and manage around 40 lambdas and 10 step functions. Monitoring for failure is conducted using Cloudwatch alarms, Ops Genie and Slack bots. Never been an issue. In our setup we are aiming for an infrastructure that is immutable and cryptological verifiable. It turned out to be bit of a challenge. :)

</comment><comment><USER30> Best to keep your workloads as small as possible, cold starts can be very bad, depending on the type of project. Been using mostly node myself, and it's worked out well.

One thing to be careful of, if you're targeting input into dynamodb table(s), then it's really easy to flood your writes. Same goes for SQS writes. You _might_ be better off with a data pipeline, and slower progress. It really just depends on your use case and needs. You may also want to look at Running tasks on ECS, and depending on your needs that may go better.

For some jobs the 5minute limit is a bottleneck, others it's the 1.5gb memory. Just depends on exactly what you're trying to do. If your jobs fit in Lambda constraints, and your cold start time isn't too bad for your needs, go for it.

<comment><USER31> > _Best to keep your workloads as small as possible, cold starts can be very > bad, depending on the type of project._

Here's a recent, interesting article on the topic that quantifies some of this: [https://read.acloud.guru/does-coding-language-memory-or- pack...](https://read.acloud.guru/does-coding-language-memory-or-package-size- affect-cold-starts-of-aws-lambda-a15e26d12c76)

<comment><USER30> Thanks for the link... interesting how quick python is to start...

</comment></comment></comment><comment><USER32> You might want to have a look at Serverless, a framework to build web, mobile and IoT applications with serverless architectures using AWS Lambda and even Azure Functions, Google CloudFunctions & more. Debugging, maintaining & deploying multiple functions gets easier.

Serverless: [https://github.com/serverless/serverless](https://github.com/serverless/serverless)

</comment><comment><USER33> Pros:

\- works as advertised, we haven't had any reliability issues with it

\- responding to Cloudwatch Events including cron-like schedules and other resource lifecycle hooks in your AWS account (and also DynamoDB/Kinesis streams, though I haven't used these) is awesome.

Cons:

\- 5 minute timeout. There have been a couple times when I thought this would be fine, but then I hit it and it was a huge pain. If the task is interruptible you can have the lambda function re-trigger itself, which I've done and actually works pretty once you set up the right IAM policy, but it's extra complexity you really don't want to have to worry about in every script.

\- The logging permissions are annoying, it's easy for it to silently fail logging to to Cloudwatch Logs if you haven't set up the IAM permissions right. I like that it follows the usual IAM framework but AWS should really expose these errors somewhere.

\- haven't found a good development/release flow for it. There's no built-in way to re-use helper scripts or anything. There are a bunch of serverless app frameworks, but they don't feel like they quite fit because I don't have an "app" in Lambda I just have a bunch of miscellaneous triggers and glue tasks that mostly don't have any relation to each other. It's very possible I should be using one of them anyway and it would change how I feel about this point.

We use Terraform for most AWS resources, but it's particularly bad for Lambda because there's a compile step of creating a zip archive that terraform doesn't have a great way to do in-band.

Overall Lambda is great as a super-simple shim if you only need to do one simple, predictable thing in response to an event. For example, the kind of things that AWS really could add as a small feature but hasn't like send an SNS notification to a slack channel, or tag an EC2 instance with certain parameters when it launches into an autoscaling group.

For many kinds of background processing tasks in your app, or moderately complex glue scripts, it will be the wrong tool for the job.

<comment><USER34> If you're already using the Node runtime, [https://serverless.com/](https://serverless.com/) fixes all of your "cons" perfectly. Terraform isn't a great fit, in my experience.

</comment></comment><comment><USER35> I started off doing manual build / deploy for a project and it was a total pain in the ass. From packaging the code, to versioning, rollbacks, deploy. Then that doesn't even include setting up API Gateway if you want an endpoint for the function.

Since then I've been using Serverless for all my projects and it's the best thing I've tried thus far. It's not perfect, but now I'm able to abstract everything away as you configure pretty much everything from a .yml file.

With that said, there are still some rough spots with Lambda:

1) Working with env vars. Default is to store them in plain text in the Lambda config. Fine for basic stuff, but I didn't want that for DB creds. You can store them encrypted, but then you have to setup logic to decrypt in the function. Kind of a pain.

2) Working within a subnet to access private resources incurs an extra delay. There is already a cold start time for Lambda functions, but to access the subnet adds more time... Apparently AWS is aware and is exploring a fix.

3) Monitoring could be better. Cloudwatch is not the most user friendly tool for trying to find something specific.

With that said, as a whole Lambda is pretty awesome. We don't have to worry about setting up ec2 instances, load balancing, auto scaling, etc for a new api. We can just focus on the logic and we're able to roll out new stuff so much faster. Then our costs are pretty much nothing.

</comment><comment><USER36> I'm reading here a lot of people jumping through some massive amounts of hoops to deal with a system that lock you down to a single vendor, and makes it hard to read logs or even read your own bill.

a few years back, the mantra was "hardware is cheap, developer time isn't". when did this prevailing wisdom change? Why would people spend hours/days/weeks wrestling with a system to save money which may take weeks, months or even years to see an ROI?

<comment><USER37> I think it's very dependent on the task. I do agree with you, building a large/high load system on lambda is bad for the reasons you're suggesting. I think a lot of the hoop jumping is due to experimentation (you won't know how much of a pain in the ass a piece of technology is until you need to deal with it).

We've mostly used it for small tasks that will get run once a day. It's been fantastic for that, as putting up a box to handle a sparsely (one or a couple of times a day) run task is a lot of work and is expensive.

</comment></comment><comment><USER38> Most of my experience mirrors that found in other comments, so here's a few unique quirks I've personally had to work around:

\- You can't trigger Lambda off SQS. The best you can do is set up a scheduled lambda and check the queue when kicked off.

\- Only one Lambda invocation can occur per Kinesis shard. This makes efficiency and performance of that lambda function very important.

\- The triggering of Lambda off Kinesis can sometimes lag behind the actual kinesis pipeline. This is just something that happens, and the best you can do is contact Amazon.

\- Python - if you use a package that is namespaced, you'll need to do some magic with the 'site' module to get that package imported.

\- Short execution timeouts means you have to go to some ridiculous ends to process long running tasks. Step functions are a hack, not a feature IMO.

\- It's already been said, but the API Gateway is shit. Worth repeating.

Long story short, my own personal preference is to simply set up a number of processes running in a group of containers (ECS tasks/services, as one example). You get more control and visibility, at the cost of managing your own VMs and the setup complexity associated with that.

<comment><USER39> > You can't trigger Lambda off SQS. The best you can do is set up a scheduled > lambda and check the queue when kicked off.

This kills me. I can't believe they haven't added this.

<comment><USER38> IIRC, it's a philosophical argument that prevents it - SQS is pull, Lambda triggers off push. They don't want to add a push mechanism to SQS, ergo they can't support Lambda.

</comment></comment></comment><comment><USER40> Pretty good actually. We started using AWS Lambda as a tool for a cron job.

Then we implemented a RESTful API with API Gateway and Lambda. The Lamdbas are straightforward to implement. API Gateway unfortunately has not a great user experience. It feels very clunky to use and some things are hard to find and understand. (Hint: Request body passthrough and transformations).

Some pitfalls we encountered:

With Java you need to consider the warmup time and memory needed for the JVM. Don't allocate less than 512MB.

Latency can be hard to predict. A cold start can take seconds, but if you call your Lambda often enough (often looks like minutes) things run smooth.

Failure handling is not convenient. For example if your Lamdba is triggered from a Scheduled Event and the lamdba fails for some reason. The Lamdba does get triggered again and again. Up to three times.

So at the moment we have around 30 Lambdas doing their job. Would say it is an 8/10 experience.

</comment><comment><USER41> For running Java in Lambda, I had to optimize it for Lambda. To decrease processing time (and in the end the bill), I got rid of all reflection for example and though twice when to initialize what and what to make static. Also, Java Cold Start is an issue. I fixed this with creating a Cloudwatch Trigger that executes the Lambda function every minute to keep it hot. Otherwise, after some minutes of no-one calling the function, it takes 10+ seconds to respond. But if you use Python for example, you don't run into this issue. I built complete backends on top of Lambda/API Gateway/Dynamo and having "NoOps" that also runs very cheap is a killer argument for me.

<comment><USER42> If you are triggering it every minute, may I ask how much you are paying for it per month . I have though about it as well as I have a java + spring + Hibernate which takes much too long to start up ; in fact the task execution time is less than the cold start time in my case.

<comment><USER41> Execution takes less than 100ms and I never exhausted my monthly free quota on Lambda, so I can't say.

</comment></comment></comment><comment><USER43> We use Node.JS lambda functions for real time image thumbnail generation and scraping needs. As well as mirroring our S3 buckets to another blob storage provider and a couple of periodic background jobs. It works beautifully. It's a little hard to debug at first but when it's set up, both pricing and reliability is really good for our use cases.

I think a lot of people try to use the "serverless" stuff for unsuitable workloads and get frustrated. We are running a kubernetes cluster for the main stuff but have been looking for areas suitable for lambda and try to move those.

<comment><USER44> Can you share your image resizing code on github? We're using thumbor on AWS but it is a huge PITA.

<comment><USER45> Setting up your own resizer using sharp[1] is pretty simple. Just make sure you install the module in a Lambda-compatible environment, so it can build its copy of libvips (native C library) correctly. I built and deployed my image thumbnailer on a CentOS VM.

[1]: [https://github.com/lovell/sharp](https://github.com/lovell/sharp)

</comment></comment></comment><comment><USER46> After multiple side projects with Lambda (e.g. image processing services), we finally implemented it on larger scale. Initially we started out without any framework or tool to help, because there we pretty much non-existent at that time. We created our own tool, and used Swagger a lot for working with API gateway (because it is really bad to work with). Over time everything smoothened out and really worked nicely (except for API Gateway though). Nowadays we have everything in Terraform and Serverless templates, which really makes your life easier if you're going to build your complete infrastructure on top of AWS Lambda and other AWS APIs. There are still a bunch of quarks you have to work with, but at the end of the line: it works and you don't have to worry much about scaling.

I'm not allowed to give you any numbers; here's an old blogpost about Sketch Cloud: [https://awkward.co/blog/building-sketch-cloud-without- server...](https://awkward.co/blog/building-sketch-cloud-without-servers/) (however, this isn't accurate anymore). For this use-case, concurrent executions for image uploads is a big deal (a regular Sketch document can easily exist out of 100 images). But basically the complete API runs on Lambda.

Running other languages on Lambda can be easily done and can be pretty fast, because you simply use node to spawn a process (Serverless has lots of examples of that).

Let me know if you have any specific questions :-)

Hope this helps.

<comment><USER47> why is API gateway so bad?

<comment><USER46> Well, using it manually is just cumbersome. API Gateway is not specifically designed for Lambda, so it has lots of settings which you would think are just default for building your API. Using it through Cloudformation or Serverless is way easier.

</comment></comment></comment><comment><USER48> Can talk only about the node.js runtime with native add-ons. Using it for various automation tasks less then 100 invocations a day where it is the most convenient solution out there for peanuts. We also use it for parsing Swagger/API Blueprint files, here we talk 200k+ invocations a day and works great once we figured out logging/monitoring/error handling and limited output (6MB). We do not use any framework because they mostly are not flexible enough but apex ([http://apex.run/](http://apex.run/)) and serves us well. We've hit couple of times some limits but as it is invocation per request only some calls failed and the whole service was unaffected. I see the isolation as a big benefit you get. One thing which sucks is that if it fails (and it is not your code) often you have no idea why and if anything can be done. We use it together with AWS API Gateway and the Gateway part is sub par. The gateway does not support correct HTTP like 204 always returns a body and god forbid if you want something else than application/json. To sum it up lambda is great with some minor warts and the API gateway is OK but can easily imagine it much better.

</comment><comment><USER49> Developing Lambda is an absolutely terrible experience. Tightening the integration between CloudFormation, API-Gateway, and Lambda would really improve the situation. For example, a built-in way to map requests/responses between API-Gateway and Lambda which didn't involve a janky parsing DSL would be pretty nice.

The strategy Lambda seems to suggest you implement for testing/development is pretty laborious. There's no real clear way for you to mock operations on your local system and that's a real bummer.

A lot of things you run into in Python lambda functions are also fairly unclear. Python often will compile C-extensions... I could never figure out if there was really a stable ABI or what I could do to pre-compile things for Lambda.

All of those complaints aside - once you deploy your app, it will probably keep running until the day you die. So that's a huge upside. Once you rake through the muck of terrible developer experience (which I admit, could be unique to me), the service simply works.

So, if you have a relatively trivial application which does not need to be upgraded often and needs very good up-time.. it's a very nice service.

</comment><comment><USER50> I've only been using it for one project right now. I made an API that I can use to push security-related events to a location that a hacker couldn't access, even if they get root on a local system. I use it in conjunction with sec (Simple Event Correlator). If sec detects something, e.g. a user login, or a package install, it'll send the event to the API in AWS Gateway + Lambda. The event then gets stored in a DynamoDB table, and I use a dashing.io dashboard to display the information. It works super well. I still need to convert my awful NodeJS code to Python, but that shouldn't take long.

I do remember logging being a confusing mess when I was trying to get this started. I feel better about the trouble I had now that I see it wasn't just me. But for a side project that's very simple to use, Lambdas have been a blessing. I get this functionality without having to manage any servers or create my own API with something like Python+Flask. Having IAM and authentication built in for me made the pain from the initial set-up so worth it.

</comment><comment><USER51> We use it with node for a bunch of things like PDF generation, asynchronous calls to various HTTP services etc. I think it's excellent.

The worst part about it by far is CloudWatch, which is truly useless.

Check out [https://github.com/motdotla/node- lambda](https://github.com/motdotla/node-lambda) for running it locally for testing btw - saved us hours!

<comment><USER45> +1 for node-lambda. It's a lot simpler than Serverless when you just want a little help with testing/deploying.

</comment></comment><comment><USER52> I've used it in production and we're building our platform entirely in Serverless/AWS Lambda.

Here are my recommendations:

1) Use Serverless Framework to manage Functions, API-Gateway config, and other AWS Resources

2) CloudWatch Logs are terrible. Auto-stream CloudWatch Logs to Elastic Search Service and Use Kibana for Log Management

3) If using Java or other JVM languages, cold starts can be an issue. Implement a health check that is triggered on schedule to keep functions used in real-time APIs warm

Here's a sample build project I use: [https://github.com/bytekast/serverless- demo](https://github.com/bytekast/serverless-demo)

For more information, tips & tricks: [https://www.rowellbelen.com/microservices-with-aws-lambda- an...](https://www.rowellbelen.com/microservices-with-aws-lambda-and-the- serverless-framework/)

</comment><comment><USER53> We're doing both stream processing and small query APIs using Lambda.

A few pointers (from relatively short experience):

\- The best UC for Lambda seems to be stream processing where latency due to start up times is not an issue

\- For user/application-facing logic the major issue seems to be start-up- times (esp. JVM startup times when doing Java or your API gets called very rarely) and API Gateway configuration management using infrastructure as code tools (I'd be interested in good hints about this, especially concerning interface changes)

\- The programming model is very simple and nice but it seems to make most sense to split each API over multiple lambdas to keep them as small as possible or use some serverless framework to make managing the whole app more easy

\- This goes without saying, but be sure to use CI and do not deploy local builds (native binary deps)

</comment><comment><USER54> I used it for converting images to BPG format and do resizing. I really enjoyed it. Basically with Docker/lambda these days I feel like the future will be 'having code' and then 'running it' (no more ssh, puppet, kuberdummies, bash, vpc, drama). Once lambda runs a docker file it might take over middle earth. These were my issues with lambda:

1\. Installing your own linux modifications isn't trivial (we had to install the bpg encoder). They use a strange version of the linux ami.

2\. Lambda can listen to events from S3 (creation,deletion,..) but can't seem to listen to SQS events WTF? It seems like amazon could fix this really easily.

3\. Deployment is wonky. To add a new lambda zip file you need to delete the current one. This can take up to 40 seconds (which you would have total downtime).

</comment><comment><USER55> For a serverless system that uses Lambda together with eg CloudFormation, Dynamo, and S3, Cognito etc - it's pretty low level and you spend a lot of time understanding, refining & debugging basic things. The end-to-end logging and instrumentation throughout the services used by your app weren't great.

Doesn't like big app binaries/JARs and Amazon's API client libs are bloated - Clojure + Amazonica goes easily over the limit if you don't manually exclude some Amazon's API JDKs from the package.

On the plus side, you can test all the APIs from your dev box using the cli or boto3 before doing it from the lambda.

Would probably look into third party things like Serverless next time.

<comment><USER56> I uploaded with Clojure 50kbyte compiled java, which have multi handler in one function. Execute times avg 15ms.

</comment></comment><comment><USER57> \- Cheap, especially for low usage.

\- Runs fast, unless your function was frozen for not enough usage or the like

\- Easy to deploy and/or "misuse"

\- Debugging doesn't really work

All in all, probably the least painful thing I've used on AWS. But that doesn't necessarily mean much.

</comment><comment><USER58> A couple of months ago, I've started using AWS Lambda for a side project. The actual functions were pretty easy to code using `nodejs` and deploying them with `serverless` but the boilerplate to opening them via an http API was the real bummer. IAMs, routing and all kind of other little things standing in the way of actual productive work. Some time after that I tried to setup GCloud Functions and to my surprise that boilerplate was minimal! Write your function and have accessible with just a couple of commands. IMHO GCloud Functions is way more developer friendly and AWS Lambda.

<comment><USER59> I work for Gcloud functions. happy to help if you need support.

</comment></comment><comment><USER60> If you need to store environment variables easily and securely take a look at EC2 Parameter Store - you can fetch the relevant parameters on startup and they are automatically encrypted and decrypted using KMS for you

</comment><comment> rlv-dan A session I remember that might be of interest:

Building reactive systems with AWS Lambda: [https://vimeo.com/189519556](https://vimeo.com/189519556)

</comment><comment><USER61> It's been great for using as 'glue' to do small tasks like clean ups in our case or other short lived minor tasks. I haven't used it for anything major though, only for minor tasks that are easier or more convenient to do with Lambda rather than a different way. The real value comes from the integration with other AWS services, for example, for developers using DynamoDb Lambdas make a lot of maintenance of records far easier with streams events.

</comment><comment><USER62> Have used it in production for > 2 years, mainly for ETL/Data processing type jobs which seems to work well.

We also use it to perform scheduled tasks (e.g. every hour) which is good as it means you don't have to have an EC2 instance just to run cron like jobs.

The main downside is Cloudwatch Logs, if you have a Lambda that runs very frequently (i.e. 100,000+ invocations a day) the logs become painful to search through, you have to end up exporting them to S3 or ElasticSearch.

</comment><comment><USER63> We run many microservices on Lambda and it has been a pleasant experience for us. We use Terraform for creating, managing environment variables, and permissions/log groups/etc. We use CodeShip for testing, and validating and applying Terraform across multiple accounts and environments.

For logging, we pipe all of our logs out of CloudWatch to LogEntries with a custom Lambda, although looking at CloudWatch logs works fine most of the time.

<comment><USER64> "looking at CloudWatch logs works fine most of the time"...are you kidding? I wanna gouge my eyes out with a rusty spoon every time I have to look at CW logs :)

</comment></comment><comment><USER65> I'm using it for a year in a half, and I'm more than happy, The cost increments when you have much load, but I am a happy user to use it for these small applications that need to be always up.

Need to say, that you should use gordon<[https://github.com/jorgebastida/gordon>](https://github.com/jorgebastida/gordon>) to manage it, Gordon makes the process easier.

Regards

</comment><comment><USER66> So far used only for toy/infrequent use cases and it works there well. E.g. Slack command, integration with different systems, cron style job.

</comment><comment><USER67> Pretty great, we're using it for resizing and serving images for our clients (large media companies, banks, etc): [https://hmn.md/2017/04/27/scaling- wordpress-images-tachyon/](https://hmn.md/2017/04/27/scaling-wordpress-images- tachyon/)

API Gateway is a little rougher, but slowly getting there.

</comment><comment><USER68> \- We use Lambda along with Ansible to execute huge, distributed ML workloads which are (completely)serverless. Saves a lot of bucks, as ML needs huge boxes.

\- For serverless APIs for querying the S3 which is a result of the above workload

Difficulties faced with Lambda(till now):

1\. No way to do CD for Lambda functions. [Not yet using SAM]

2\. Lambda launches in its own VPC. Is there a way to make AWS launch my lambda in my own VPC? [Not sure.]

<comment><USER69> 2\. You can! Although it's a bit tedious. You need to also ensure the function has EIP allocation permissions.

</comment></comment><comment><USER70> We have moved our database maintenance cron jobs to Lambda as well as the image resize functionality. General experience is very positive after we figured out hot to use Lambda from Clojure and Java. People worried about JVM startup times: Lambda will keep your JVM up and running for ~7 minutes after the initial request and you can achieve low latency easily.

</comment><comment><USER71> We use Lambda extensively at [https://emailoctopus.com](https://emailoctopus.com). The develop-debug cycle takes a while, but once you're up and running, the stability is hard to beat. Just wish they'd raise that 5 minute execution limit so we can migrate a few more scripts.

</comment><comment><USER72> It's good. We're using it for a ton of automation of various developer tasks that normally would get run once in a while (think acceptance environment spinup, staging database load, etc.).

It fails once in a while and the experience is bad, but that's mostly due to our tooling around failure states instead of the platform itself.

</comment><comment><USER73> Hey everyone, I'm Daniel Langer and I help build lambda monitoring products over at Datadog. I see lots of you are unhappy with the current monitoring solutions available to you. If anyone has thoughts on what they'd like in a Lambda monitoring service feel free to email me at firstname.lastname@example.com

<comment><USER74> I spent a while talking to the Datadog guys at the AWS Sydney summit in April and while the product was compelling, nobody could give me a straight answer on pricing for lambda - all the pricing is given in terms of per host. So I'd have to say the main thing I'm after is pricing transparency...

</comment></comment><comment><USER75> Been using it for about 6 months with Serverless for Node API endpoints and it's great so far!

The only negatives are: \- cold start is slow, especially from within a VPC \- debugging/logging can be a pain \- giving a function more memory (~1GB) always seems to be better (I'm guessing because of the extra CPU)

</comment><comment><USER76> \- When using with API Gateway the API response time is more than 2-3 seconds for a NodeJS lambda, for Java it will be more. \- Good for use cases for example \-- cron - can be triggered using Cloudwatch events. \-- Slack command bot (API Gateway + Lambda) the only problem is timeout.

</comment><comment><USER77> I'm having around 4 mln lambda executions per month, mostly on data processing and I'm happy in overall with performance and easy of deployment. Debugging is hard, frameworks are still very mature. I use AWS SDK and C# and I'm having quite good experience.

</comment><comment><USER78> Lots of great comments here. I'd like to add that being limited to 512mb of working disk space at /tmp has been a stumbling block for us.

Would be really great to have this configurable along with CPU/memory.

Additionally being able to mount and EFS volume would be very useful!

<comment><USER79> Is there any reason why you can't use S3?

<comment><USER78> Sure, and that's what I do most of the time.

However, if the S3 keys are larger than ~500mb, then it's not possible to process them in any way with Lambda since there isn't enough "scratch space" available in /tmp.

I was suggesting EFS support merely because it would allow access to arbitrarily large amounts of "local" disk to work with...

<comment><USER79> didn't know you need scratch space to work with S3 in lambda? What about something like s3fs/goofys?

</comment></comment></comment></comment><comment><USER80> \- There is a surprisingly high amount of API gateway latency

\- The CPU power available seems to be really weak. Simple loops running in NodeJS run way way slower on Lambda compared to a 1.1 GHz Macbook by a significant magnitude. This is despite scaling the memory up to near 512mb.

\- Certain elements, such as DNS lookups, take a very long time.

\- The CloudWatch logging is a bit frustrating. If you have a cron job it will lump some time periods as a single log file, other times they're separate. If you run a lot of them its hard to manage.

\- Its impossible to terminate a running script.

\- The 5 minute timeout is 'hard', if you process cron jobs or so, there isn't flexibility for say 6 minutes. It feels like 5 minutes is arbitrarily short. For comparison Google Cloud Functions let you work 9 minutes which is more flexible.

\- The environment variable encryption/decryption is a bit clunky, they don't manage it for you, you have to actually decrypt it yourself.

\- There is a 'cold' start where once in a while your Lambda functions will take a significant amount of time to start up, about 2 seconds or so, which ends up being passed to a user.

\- Versions of the environment are updated very slowly. Only last month (May) did AWS add support for Node v6.10, after having a very buggy version of Node v4 (a lot of TLS bugs were in the implementation)

\- There is a version of Node that can run on AWS Cloudfront as a CDN tool. I have been waiting quite literally 3 weeks for AWS to get back to me on enabling it for my account. They have kept up to date with me and passed it on to the relevant team in further contact and so forth. It just seems an overly long time to get access to something advertised as working.

\- If you don't pass an error result in the callback callback, the function will run multiple times. It wont just display the error in the logs. But there is no clarity on how many times or when it will re-run.

\- There aren't ways to run Lambda functions in a way where its easy to manage parallel tasks, i.e to see if two Lambda functions are doing the same thing if they are executed at the exact same time.

\- You can create cron jobs using an AWS Cloudwatch rule, which is a bit of an odd implementation, CloudWatch can create timing triggers to run Lambda functions despite Cloudwatch being a logging tool. Overall there are many ways to trigger a lambda function, which is quite appealing.

The big issue is speed & latency. Basically it feels like Amazon is falling right into what they're incentivised to do - make it slower (since its charged per 100ms).

PS: If anyone has a good model/providers for 'Serverless SQL databases' kindly let me know. The RDS design is quite pricey, to have constantly running DBs (at least in terms of the way to pay for them)

<comment><USER81> RDS always running was just recently fixed...

[https://aws.amazon.com/about-aws/whats-new/2017/06/amazon- rd...](https://aws.amazon.com/about-aws/whats-new/2017/06/amazon-rds-supports- stopping-and-starting-of-database-instances/)

</comment></comment><comment><USER82> \- Decouple lambdas with queues and events, SQS, SNS and S3 events are your friends here

\- Use environment variables

\- Use step functions to create to create state machines

\- Deploy using cloudformation templates and serverless framework

<comment><USER38> Since you can not trigger Lambda with SQS, I don't recommend using Lambda with SQS. You end up having to use some other tool to trigger the Lambda invocation, and polling SQS from there.

Better to use a stream that can trigger Lambdas natively - like SNS or Kinesis.

</comment></comment><comment><USER83> In the end it's only the web server that is serverless, you still need other servers depending on your use case, and hey, web servers aren't that hard to run anyway.

</comment><comment><USER84> We had a bad experience: we accidentally made an error in one function that got called a lot, which blocked other functions from runnning. Yay for isolation!

</comment><comment><USER85> I'd like to use Lambda@Edge to add headers to my CloudFront responses. Does anybody have any idea when this might be released from preview?

</comment><comment><USER86> Can anyone speak to continuous deployments with Lambda, where downtime is not an option? Is it possible to run blue green deployments?

<comment><USER10> If downtime is an not an option, Lambda is not the solution. Amazon will automatically shut down instances of your Lambda if they haven't been used (~5-15 minutes), and starting a fresh Lambda has noticeable latency. So, some people have resorted to periodically querying their Lambda to prevent this. However, occasionally Amazon will reset all Lambdas, which will force the hard restart.

<comment><USER86> I'm aware of the increased startup latency for functions haven't been recently used, but that's not the same as downtime or dropped connections.

</comment></comment></comment><comment><USER87> I tried using Lambda, but need to set up the API gateway before using as well. Painful logging and parameter forwarding.

<comment><USER81> That makes no sense, API Gateway can proxy requests directly to lambda, and in C# there's Nancy / Web API middleware.

</comment></comment><comment><USER83> Lots of people saying the API gate away is hard.

 _You don 't need to use the API gateway_.

Just talk direct to Lambda.

</comment><comment><USER88> Any comparisons to GCE and Azure's offerings for those who have used both?

</comment><comment><USER89> I've only played with it as opposed to deploying Prod but guve Azure Functions a try too [https://azure.microsoft.com/en- us/services/functions/](https://azure.microsoft.com/en-us/services/functions/)

</comment><comment><USER90> Works terribly. It's basically a thin wrapper around a shoddy jar framework. All the languages supported are basically shit-farmed from the original Java one. The Java one is the only one that works half decently.

<comment><USER81> This sounds like you touched it for all of 30 seconds hated it and formed an opinion.

I use both NodeJS and C# lambda's without issue. The support is really good.

Debugging experience isn't great but aside from that it's fast and easy to use.

C# lambda's can call RDS and respond back in ~5ms...

(before anyone calls me out on the 5ms...)

[http://www.philliphaydon.com/2017/05/10/part5-configuring- th...](http://www.philliphaydon.com/2017/05/10/part5-configuring-the-vpc-so- we-can-call-the-database/)

Last image, I state 2-3 second startup and then 4ms response on a call to the database.</comment></comment>


Cache Sizes with CPUID on Intel CPU's - <AUTHOR> http://blog.melkerlitsgard.se/2016/05/12/cache-sizes-with-cpuid/ <comment><USER2> I don't get it. The code doesn't even compile. Both code snippets have undefined variables.

<comment><USER3> Just declare cpu_info as int[4] and it should run.

Basically on x86 CPUs you can use the cpuid instruction to get information about the CPU and the cache: [http://www.sandpile.org/x86/cpuid.htm](http://www.sandpile.org/x86/cpuid.htm)

The code in the first snippet uses a compiler intrinsic to execute this instruction and calculates the cache size from the result. In the second snippet the assembler code is written by hand instead.

<comment><USER2> Yes, that was pretty clear, but in the D version that's not enough to make it compile, there is also the missing functionID which apparently needs to be primed with the correct value to make things work. No idea what that value should be though.

It's not that I thought that this information was difficult to look up, rather it seemed very odd information to be missing from a link that was submitted as useful/interesting.

<comment><USER3> functionID is what's called "standard level" in the link I posted. So it should be 4 (you can also see that in the C version btw.).

I agree that the example isn't very well tested - the author probably pulled it out of some existing code and wasn't very careful.

</comment></comment></comment></comment><comment><USER4> on Debian Linux, the commandline programs 'cpuid' and 'x86info' from packages of the same name show this information; their source might also be worth reading to understand how to obtain and interpret this information in practice.

<comment><USER5> lscpu (from the preinstalled util-linux package) works too. And now that Windows comes with the Ubuntu Bash …

<comment><USER6> Might as well just read from /proc/cpuinfo</comment></comment></comment>


Does Digg Have a Secret Co-Founder in the Attic? - <AUTHOR> http://blog.wired.com/business/2008/11/kevin-rose-is-k.html

<comment><USER2> Not news to us since ojbyrne posts here, but it's nice to see Owen getting some of the recognition he deserves.

</comment><comment><USER3> So what is a co-founder? Kevin came up with the concept and paid Owen to implement it. Owen may have been there since the beginning and undoubtedly had an influence in its direction, but a "founder"? Dunno seems to be a gray area.

<comment><USER4> I've never been clear on that myself. So I'll suggest a simple test - let's look at another company from the same keiretsu - pownce.com. Leah Culver has been a salaried employee since the beginning, and is described as a "founder."

What's the difference? What makes a "founder?" Significant equity? I pass. At some point in time others in the enterprise thought of you as a "founder?" I pass. At some point later on people decided you weren't a founder? I fail on that one.

<comment><USER5> Are you a hot young blonde girl? I think that's the litmus test.

<comment><USER4> What I find fascinating is that I've been following Kevin's advice about PR - tweet, comment, etc. etc. It works, though it takes some basic notoriety to start with, and some persistence.

<comment><USER5> Well he also lives in SF where every other woman working in PR has the hots for him.</comment></comment></comment></comment></comment>


Tech Crunch interview with Jordan Rudess - <AUTHOR> http://www.crunchgear.com/2009/08/28/interview-jordan-rudess-from-dream-theater/

<comment><USER2> It's very neat to see some musicians (especially those as talented as Rudess is) down in the trenches working on things like this. A few years back, the company I worked for was a sponsor of BT's This Binary Universe tour, and he came in to talk to us about the technology used. He and his team of developers hand-wrote a suite of software for sequencing and editing audio in a native surround sound format. He talked about the C++ used to develop it the same way I've seen so many musicians talk about their guitars. Quite an amazing experience.</comment>


What Was Once Hailed as First U.S. Offshore Wind Farm Is No More - <AUTHOR> https://www.bloomberg.com/news/articles/2017-12-01/cape-wind-developer-terminates-project-opposed-by-kennedys-koch <comment><USER2> Man, sometimes I feel like the Kennedys are America's most hypocritical family.

Edit: BTW, I go by wind farms all the time on my way through beautiful Cle Elum Washington and they do not spoil the view in the slightest.

They are not nearly as jarring as the ugly houses people are building in the upper Madison County in Montana where my family lives.

<comment><USER3> > Man, sometimes I feel like the Kennedys are America's most hypocritical > family.

I don't know what this means. Are you referring to actions taken by other Kennedy family members? If so, why should the actions of one's relatives fall exactly in line with one's own legacy, lest their family be deemed hypocritical? That seems like an awfully high bar that presupposes the family is a single moral and ethical unit.

<comment><USER4> How about when Ted Kennedy stopped Richard Nixon's healthcare plan.

[https://www.google.com/amp/s/www.bostonglobe.com/opinion/201...](https://www.google.com/amp/s/www.bostonglobe.com/opinion/2012/06/22/stockman/bvg57mguQxOVpZMmB1Mg2N/amp.html)

<comment><USER5> You mean, when Nixon introduced a much weaker healthcare bill right after Kennedy had managed to broker a bipartisan bill proposing a real reform? I don't see anything hypocritical about refusing to support a bill designed to keep the current broken system running and squash actual reform.

At least, that's how it looks to me as a complete outsider.

<comment><USER4> well, that's not what the article says that I posted:

"Kennedy said later that walking away from that deal was one of the biggest mistakes of his life."

"Over time, Kennedy realized his own plan couldn't succeed. Opposition from the insurance companies was too great. So Kennedy dispatched his staffers to meet secretly with Nixon's people to broker a compromise. Kennedy came close to backing Nixon's plan, but turned away at the last minute, under pressure from the unions. Then Watergate hit and took Nixon down."

<comment><USER5> I read that, and I don't think it contradicts my view.

Kennedy introduces a real reform bill; Nixon introduces a weak, "compromise" bill so the opponents of real reform (like the insurance companies) can use to oppose it while avoiding the wrath of the people who understand the status quo is terrible, by looking like they also want changes. Kennedy, defeated, judges whether accepting Nixon's bill - despite its intent - is still worth it, or whether the unions are right that it imposes too great of a cost on employees (25% of premiums). He makes a decision, then later he regrets it.

</comment></comment></comment></comment></comment></comment><comment><USER6> Kind of reminds of this one:

[https://www.reuters.com/article/us-usa-fracking- tillerson/ex...](https://www.reuters.com/article/us-usa-fracking- tillerson/exxon-mobil-ceo-welcomes-fracking-but-not-water-tower-in-his- backyard-idUSBREA1P24O20140226)

Tillerson, former Republican heavyweight Dick Armey and other residents of a ranch-filled suburb of Bartonville north of Dallas filed suit in 2012 seeking to block construction of the 160-foot-tall (49-meter-tall) water tower, arguing it would be an eyesore.

The suit, filed in Denton County District Court, also noted that the tower could encourage the town of Bartonville to sell "water to oil and gas explorers for fracking shale formations leading to traffic with heavy trucks... creating a noise nuisance and traffic hazards."

</comment><comment><USER7> A bit of a bummer to hear, especially when Massachusetts is shutting down its only nuclear power plan in 2019. The plant produces the vast majority of the State's renewable energy and about 15% of its total power. It doesn't sound like there are real replacements lined up.

<comment><USER8> How does a nuclear plant produce renewable energy?

<comment><USER9> I just assumed they mean _clean energy_.

<comment><USER10> I just assumed that they mean carbon-poor electricity.

</comment></comment></comment></comment><comment><USER11> This is some really flagrant and kind of reprehensible NIMBYism.

</comment><comment><USER12> I just don't get all the arguments that say wind farms are ugly. I just don't.

<comment><USER13> Some friends and I visited an wind farm just as a place to fly a kite. The area was rolling green hills, a few cows and these massive, clean, white towers up into the sky. It honestly struct me as a scene from an idyllic future sci-fi movie. It was like walking around in someone's CG desktop background.

<comment><USER12> In my hometown we built a wind turbine on the top of a mountain that overlooks the city. On clear days you can see the turbine poking out on top of the mountain and it looks absolutely gorgeous, like we're an advanced species intelligently harvesting energy from the planet. It's breathtaking in a way that's hard to describe.

</comment></comment></comment><comment><USER14> That wind farm could have eventually provided enough power for a million people. As much as the specific loss of renewable energy is detrimental in this case, the loss of another powerful demonstration in the US of offshore wind farm potential is probably just as great.

</comment><comment><USER15> America has the best leadership money can buy.

</comment><comment><USER16> Are there areas off the coast of Massachusetts where this style of offshore wind farm can be implemented that aren't proximal to the most popular and picturesque vacation destinations in the area? I'm sure most people can get behind the idea of offshore wind but this wasn't the best thought out location for this project.

<comment><USER17> This has nothing to do with the landscape. I've driven by the pilot windmills. You can barely see them. This is about filthy rich white hypocrites throwing a tantrum over nothing. Most people in Massachusetts (or America for that matter) would probably have a hard time affording to vacation anywhere near these things.

<comment><USER18> What does their being white have to do with it?

<comment><USER17> That's literally almost all you'll find on either of the islands, to the point that it sticks out if there's a POC there. I'm a white upper middle class guy myself, but let's call a spade a spade. They tried to build this in the rich white guy's back yard and they didn't like it.

Edit: These are also the same people that claim to be progressives and for the POC(I'm liberal) in politics but when it comes to actually doing something that would have cleaner air, their first reaction is to NIMBY it. These are exactly what these people are. It's a definition.

<comment><USER19> As a relatively poor, white eastern European I feel somewhat offended that I'm thrown in the same bin as these people solely because of something over which I have no control - the tone of my skin.

</comment></comment></comment></comment></comment><comment><USER20> By comparison, the UK has over 11 gigawatts of offshore wind capacity...

<comment><USER21> According to [0] it was 15 gigawatts in 2016 Q3. Very nice wind at the British isles.

That's nominal capacity, rate of generation appears to be 24 % of nominal; what can effectively be utilized is then below that.

[0] [https://en.wikipedia.org/wiki/Wind_power_in_the_United_Kingd...](https://en.wikipedia.org/wiki/Wind_power_in_the_United_Kingdom)

<comment><USER22> Around 2.3GW now by wind [http://www.gridwatch.templar.co.uk/](http://www.gridwatch.templar.co.uk/)

It is weather dependant but that number grows depending on the day

</comment></comment></comment><comment><USER23> It died at the hands of The Kennedys and the Kochs. An unholy alliance of progressives and conservatives sunk the project. Here we see that energy policy trumps politics when it comes into conflict with people's personal interests.

On the other hand, the industry learned so long as the turbines are out of sight, they are out of the NYMBY's minds.

<comment><USER24> It is disgusting that any individual has that sort of power.

Being in politics is supposed to mean being a public _servant._

Well okay, how much power do you think Senators give their servants? They probably don't think of their housekeepers/sitters/chefs/etc's opinions at all. If their dog-walker objects to the dog that they get, what's going to happen? The dog sure isn't going to change.

<comment><USER25> We call it "servant", but in fact still treat them like royalty. We never fully made the transition from royalty to servant. Why do so many governments put their high officials up in special house/palaces? We don't do that for any other jobs. It would be much healthier to treat elections like a shared effort to hire someone to do a job.

<comment><USER24> Terry Pratchett has a lot of good aphorisms, but one that stuck with me is some etymology from a cop and a Machiavellian dictator:

> "It's all for the good of the city, sir. Do you know where the word > 'policeman' comes from? It means 'man of the city', sir. From the old word > _polis._ "

> ...

> "You're a man interested in words, captain. I'd just invite you to consider > something your predecessor never fully grasped."

> "Sir?"

> "Have you ever wondered where the word 'politician' comes from?" Said the > Patrician.

These roles have a very specific purpose, and that purpose is service. They've veered sharply away from it, but that is still their job. Also, I think in real life the root is 'politia' or 'politika' or something, but the definition is similar.

And another word from that book: 'polite' apparently came from 'behavior befitting someone living in a city.' I guess social norms maybe became more important with the density?

<comment><USER25> This is all from ancient Greek where the city ("polis") was the state.

These terms were thus all created in the first democracy we know of. However, that was then followed by millennia of monarchies. Modern democracies came into existence in an environment where monarchies were still the norm and it shows. I find it reminiscent of the fact that Christians celebrate Christmas which is supposedly a Christian holiday by following all kinds of pagan rituals like the entire Christmas tree thing.

</comment></comment></comment></comment></comment><comment><USER26> Good. It was a corrupt boondoggle operation that was going to cost everyone a heck of a lot of money.

And I say this as someone who would like to see global warming hurry up and wash eastern MA into the Atlantic ocean.

Also, it takes a special kind of stupid to think you can take on rich and very well connected people and win.

>Lawsuits piled up, delaying the project. Cape Wind missed a series of contractual milestones, prompting National Grid Plc and Northeast Utilities' NSTAR unit to cancel power-purchase agreements in early 2015. At the time, analysts declared the project all but dead.

I forget, is that the one where people were supposed to pay more than the already sky-high electricity rates for Locally Grown(TM) Non GMO(TM) Green(TM) energy or was that a different one?

</comment><comment><USER27> [http://www.cc.com/video-clips/nmuqcf/the-daily-show-with- jon...](http://www.cc.com/video-clips/nmuqcf/the-daily-show-with-jon-stewart- jason-jones-180---nantucket)

</comment><comment><USER28> I automatically closed the tab when the video started playing automatically.

<comment><USER29> Same here. Not sure why you are being downvoted.

<comment><USER28> Well, my comment isn't adding anything useful to the conversation, hence the downvote. I still wanted to point it out though :).

</comment></comment></comment><comment><USER30> Have any of you lived near a wind farm ? A few years ago I camped near one that had 5 wind mills, and it was 7 miles (11 k?) away, It was like I was sleeping with the loudest snorer you can imagine. I slept very little.

Maybe when located out in the ocean the sound would blend, but if on land I would not want one anywhere near where I live.

Anyway I heard they are making great strides with solar and even tidal power, I doubt those will keep you awake.

<comment><USER31> > Have any of you lived near a wind farm ? A few years ago I camped near one > that had 5 wind mills, and it was 7 miles (11 k?) away, It was like I was > sleeping with the loudest snorer you can imagine. I slept very little.

That really surprises me.

I've got a dozen turbines less than 10km away literally facing my windows (I can see them unless it's really foggy) and I never heard them, despite having real trouble sleeping when there's any sort of noise. The turbines are half a klick from a small town too, they'd have been burned down a long time ago if they could be heard from 10.

<comment><USER30> > That really surprises me.

well maybe it was the terrain, it is a mountainous and maybe the shape caused come kind of reverb. I know I was warned by my camping neighbors about the sound and I thought "how bad can it be".

But it was a rough night with little sleep due to the sound.

</comment></comment></comment><comment><USER32> They're putting these windmills all over West Virginia - its a shame really. They really spoil the views and kill lots of birds. It's another case that shows the difference between what the little people have to deal with vs the elites.

<comment><USER33> I was hoping this was a parody, but I fear it's not.

The impact on views of wind turbines is minimal. Even if you don't like 'em, I'm sure you'll acknowledge that some minor inconvenience is better than long- term environmental damage.

Also, wind turbines dont really do that much damage to bird populations: [https://science.howstuffworks.com/environmental/green- scienc...](https://science.howstuffworks.com/environmental/green-science/wind- turbine-kill-birds.htm)

<comment><USER32> Not everyone agrees with you:

[http://savetheeaglesinternational.org/new/us-windfarms- kill-...](http://savetheeaglesinternational.org/new/us-windfarms- kill-10-20-times-more-than-previously-thought.html)

My point was that the elites of Martha's Vineyard were able to kill the project, and at least part of that was based on what they look like. Windmills in the distant ocean view are much less impacting of views than ones prominently arrayed across ridge lines.

It appears there's one rule for little people and other for elites.</comment></comment></comment>


Some companies in Sweden are testing a six-hour work day - <AUTHOR> http://www.fastcoexist.com/3051448/why-sweden-is-shifting-to-a-6-hour-work-day <comment><USER2> As a swede I think the title is highly misleading. Sweden is not shifting to a 6-hour workday. Sure there are a few experiments but the baseline is that sweden (which has a huge public sector) cannot afford it without raising taxes further.

The current forecasts for the future indicates that taxes has to be raised a lot just to maintain the current standard. I would be very surprised if the 6 hour workday would be prominent here in the future.

<comment><USER3> As an American, I'm interested in knowing why it would require Sweden to raise taxes. Since you mentioned a huge public sector, is it a case of needing extra workers to work shifts so that pubic services are run at the usual hours?

I imagine private companies would be able to get away with it, as long as all their competitors also switch to six-hour workdays.

<comment><USER2> From just a quick googling around I've seen estimated costs for a 30 hour work-week the public budget between 86 [1] - 236 [2] billion SEK/year. The total budget last year year was 862 billion. Somewhere that money has to come from

Some Swedish links about it: [1]: [http://www.etc.se/inrikes/sa-tycker- oppositionen-om-sextimma...](http://www.etc.se/inrikes/sa-tycker-oppositionen- om-sextimmarsdagen)

[2]: [http://www.expressen.se/nyheter/val2014/sa-mycket-kostar- fis...](http://www.expressen.se/nyheter/val2014/sa-mycket-kostar-fis-forslag- att-genomfora/)

<comment><USER2> Here's some more facts about the public sector. Translated from: [http://www.scb.se/Statistik/_Publikationer/OE0903_2014A01_BR...](http://www.scb.se/Statistik/_Publikationer/OE0903_2014A01_BR_OE06BR1401.pdf)

Different measures of government size, 2012. Percentage of GDP.

Public authorities' share of the total value added: 18%

Public sector consumption and investments: 30%

Employed in the public sector: 28%

Employed in the public sector and public enterprises 33%

General government expenditure 51%

General government revenue 50%

</comment></comment></comment></comment><comment><USER4> This hasn't been in the news in Sweden a lot, there is nothing "official" going on and I doubt that a fraction of a percent of those with a 40h week today will have anything else in a year or two. The headline (both in the source and here) should read "a couple of companies and public employers in Sweden are testing a 6-hour work day".

Nothing to see here.

<comment><USER5> And it might be worth knowing that the two mentioned companies (Filimundus AB, M Brath AB) in the article - each have about 10 employees, according to 2014 official records.

I'd say that it's quite hard to extrapolate that the whole country is going the same way as these two companies.

<comment><USER4> Parts of some large municipalities and hospitals (City of Gothemburg, for example) are trying it at a much larger scale than these companies, and the results will be much more interesting. It's still not widespread though.

</comment></comment></comment><comment><USER6> It is not mentioned in the article but a many places has also opted to implemented a 2 team shift. This mean that instead of having production during 8 hrs each day and seeing significant drop in production during the later hours, they instead got 12 hr of very effective workers. One hospital that did this has reported significant increase in number of surgeries, decrease in mistakes, few'er sick days, lower turnover rate, and easier time to recruit. The doubling in employees are paid not by taxes but instead with higher efficiency, and as such we might not need to build new hospitals in a while if this was implemented as a national policy.

</comment><comment><USER7> I've found that working 8 or 9 months during the year then taking 3 or 4 months to look for the next contract or FT job helps me to re-sync and avoid burn-out (at least, I don't think I'm burned out - that's a good sign right!).

<comment><USER8> way to go! I try the same, but when I stop working and chill for a while, I start to feel guilty for not making more money despite I could.

Maybe a personal year-income-threshold would be a good idea. Once that's earned, just relax. Makes more sense in countries with progressive income tax rate.

<comment><USER9> > Maybe a personal year-income-threshold would be a good idea. Once that's > earned, just relax. Makes more sense in countries with progressive income > tax rate.

Isn't a progressive income tax usually applied monthly? In that case, not working for some calendar months is hurting your income.

At least where I am, we have progressive income tax, and if I did this I would end up paying more in taxes for the same amount of work, as for 3 months I would not use any of my lower-tier income tax, and the rest of the year I would be paying more to the higher tiers than if I balanced things evenly over the year.

I guess you could make a deal with your employer to split your payments across the inactive months, not sure if there are some legal issues with that.

One of the many ways tax systems can create weird incentives, as this actually sounds like a pretty nice arrangement.

<comment><USER10> The US withholds taxes based on the amounts in the paycheck, but the actual tax is figured on annual income, so you either get the money back or given a predictable annual income, adjust the withholding (there is some risk of getting penalized for owing too much, so the predictable part matters).

<comment><USER7> Correct, this is in the US and in fact to the parent, we found we end up not with much less in our account making 80K instead of 120K. I suppose maybe if you go above 150K it might be better, not sure about that though.

</comment></comment></comment></comment></comment><comment><USER11>
      Filimundus switched to a six-hour day last year, and says that
      the change hasn't really made a major difference in how people
      work. The leadership team just asked people to stay off social
      media and personal distractions, and eliminated some standard
      weekly meetings.

Might work for a company with a good culture, but betting on people to be grateful and voluntarily improve their work ethic works only as long as all the others still have the 8h workday.

</comment><comment><USER12> Can't wait till this happens in North America. I hope we realize shorter work days and basic income in the next couple of decades. Then we will truly have a golden age.

<comment><USER13> I fully expect the US to be the very last to switch.

<comment><USER14> I'm kind of Ok with letting other countries try this one out first. I'm receptive to the idea of basic income, but the potential unintended consequences seem terrifying.

<comment><USER15> If anyone is to try this, the states are ideal candidates. People can (and do) vote with their feet when states implement negative policies, which is one reason they are "laboratories of democracy". In addition, it is clear that they have the consitutional authority to do this (while the federal government does not).

<comment><USER14> It feels like something that would fail if only one state did it. A lot of low income people would move to that state for basic income and the high income people seeing massive tax increases would leave. Then the system would collapse.

At a country level it is much more difficult for people to come and go and countries have some control over who does come and go. It would be better tried by some European country.

</comment></comment></comment></comment></comment><comment><USER16> This is a great idea - people waste so much time at work that if they only worked 6 hours they might get more done, and have a better quality of life, leading them to be more productive when they are there.

</comment><comment><USER17> As an expat.living in Sweden with kids, there's a lot of social pressure to not leave your kids in childcare longer than 8 hrs. If both caregivers don't have flexitime, 6 hrs would be very attractive to IT workers here. Both male and female.

</comment><comment><USER18> Standardizing work always leads to problems.

Some individuals need 3 hours, others 12 hours. The only question is whether the job is done.

<comment><USER19> And sometimes I need 6 hours in the office to finish the day in a 2 hour blaze of glory

</comment></comment><comment><USER5> It might be worth noting that the two companies (Filimundus AB, M Brath AB) mentioned in the article both have about 10 employees each, according to official records from 2014. Both companies have been running since approximately 2011.

I think it's pretty safe to say that "Sweden" is not moving to a 6 hour work day yet. This is two, incredibly small companies that are doing so.

If it was Ericsson AB or any of the industrial companies (Sandvik, SCA, Volvo, SSAB ie Swedish Steel) however...

</comment><comment><USER4> One of the reasons why Swedes find it most necessary to push for this (they don't to any large extent, but when countries do, Swedes will be among the first) is that we don't do 9-5, we do 8-5 (white collar) or 7-4 (blue collar) because we have 45-60 minutes of unpaid lunch break. As a swede, the term "9-5" sounds like a holiday really!

</comment><comment><USER20> While I agree that an arbitrary eight-hour work day may not suit every task the best, I don't see why switching it to six hours is any less arbitrary.

As a self-employed person my pay _directly_ correlates to the amount of hours I grind away. I actively try to make my working time more efficient, but cutting down on hours wouldn't get me anywhere.

Why not advocate that shifts be of a length required by the task alone and not a preset amount of time each day?

<comment><USER21> Because it's mentally taxing to work for eight hours straight for many if not most people. Knowledge work ([http://calnewport.com/blog/2012/11/21/knowledge- workers-are-...](http://calnewport.com/blog/2012/11/21/knowledge-workers-are- bad-at-working-and-heres-what-to-do-about-it/)) does not scale linearly with time, even over the course of a workday.

<comment><USER20> I find I work best in two sprints less than 6 hours, I totally get that long sprints reach a point of diminishing returns. My question is Why is six hours so much better than abandoning shift-based timing for something like task- based timing instead and cutting out all that extra time, not just reduce it.

</comment></comment></comment><comment><USER22> I would like to try out a 10h workday with 4days/week and a 3 day weekend. Some studies say a 3 day weekend makes you happier and more productive.

<comment><USER23> why 10h? go ahead with 4 days but keep 8h.

<comment><USER24> 1\. Employers want to get 40h work out of every employee

2\. Employees don't want to take a 20% cut to their salary

<comment><USER23> It's all a matter of negotiation, isn't it. You never get what you never ask for.

<comment><USER22> No success yet, but if you find a company that pays a CTO with 4x8h give me a call.

</comment></comment></comment></comment></comment><comment><USER25> I probably work about 10 hours per week, offsetting that by basically being available 24-7 if needed. Luckily, that's very rare. I guarantee my workload and quality of work excels my coworkers who are in the office 40+ hours a week. The 40 hour work week is totally ineffective for most industries but, alas, America would never move away from it.

</comment><comment><USER23> I personally always negotiate for 4days a week. Works wonderfully for me.

</comment><comment><USER26> ..and they established they were 25% less productive.

</comment><comment><USER24> The majority of people have children. Sucks for you, but I'd rather it suck for you and the rest of the few asshole asocial hermits, than suck for all of us. Though if you like money so much, maybe we could pass in a law that you can trade in your fertility for a tax cut.

<comment><USER27> I second the other comment. Fuck you, just, fuck you.

<comment><USER28> Please follow the HN guidelines and remain civil, even when the other person has not.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

[https://news.ycombinator.com/newswelcome.html](https://news.ycombinator.com/newswelcome.html)</comment></comment></comment>


Linux 5.6 - <AUTHOR> https://lwn.net/Articles/816216/ <comment><USER2> Kernelnewbies provides a nice community changelog[0]. It is a skeleton as I post this, but it'll be useful later on, as is the case with previous Linux releases.

[0]: [https://kernelnewbies.org/Linux_5.6](https://kernelnewbies.org/Linux_5.6)

</comment><comment><AUTHOR> This is the release that brings us Wireguard on mainline! [https://lists.zx2c4.com/pipermail/wireguard/2020-March/00520...](https://lists.zx2c4.com/pipermail/wireguard/2020-March/005206.html)</comment>


Delete your list of startup ideas, they are holding you back - <AUTHOR> http://maxkle.in/delete-your-list-of-startup-ideas-they-are-holding-you-back/ <comment><USER2> who is max klein and what has he done?</comment>


Xkcd launches wikipedia fundraiser, xkcd-style - <AUTHOR> http://xkcd.com/1193/

<comment><USER2> As if watching "Time" wasn't enough, we've now got another comic updating every couple of minutes... Thanks Randall.

</comment><comment><USER3> Unfortunately the mobile versions renders broken in the iPad. :-/

Now I'll have to find a regular computer ASAP.

</comment><comment><AUTHOR> Mouse over the panels.

</comment><comment><USER4> Here's the thread on the xkcd forums about it: [http://forums.xkcd.com/viewtopic.php?f=7&t=101147](http://forums.xkcd.com/viewtopic.php?f=7&t=101147)

They have some cached images of the previous sizes of the dog, among other things.</comment>