Cadence: Uber's Workflow Orchestration Engine - <AUTHOR> https://github.com/uber/cadence <comment><USER2> Cadence looks like an OSS version of Amazon Simple Workflow (SWF) service. The author used to work on SWF at AWS afaik.

I'm a heavy SWF user at work for managing complex data pipelines. SWF requires an important conceptual and tooling effort in the beginning, but it gets reimbursed if you use it a lot.

As for other comments mentioning Airflow: the programming model is quite different , since Airflow as far as I understand forces to provide a DAG of tasks upfront. SWF (and Cadence?) doesn't, it coordinates the work of Deciders and Activity Workers and only acts as a source of truth for the state of the workflow (+ distribute task in a unique manner to many long-polling workers). As a result you don't declare anything upfront and can have deciders take dynamic decisions along the way, which is really nice when you want very dynamic logic for your workflows (e.g. dynamic partitioning of tasks, decisions depending on external factors, etc.).

I'd love to have Maxim insights about how Cadence compares to SWF, and what would be the reasons/challenges behind migrating from SWF to Cadence for SWF users (except that SWF is basically stale for 4+ years and rigged with arbitrary limits)

<comment><USER3> Cadence vs SWF

Cadence was conceived and is still led by the original tech leads of the SWF.

SWF had no new features added for the last 5 years. Cadence is open sourced and is under active development.

Cadence was initially based on SWF public API. It uses Thrift and TChannel for communication and SWF uses AWS version of REST. Currently the API is not compatible with SWF as Cadence added a large number of new features and deprecated a few problematic ones. We are planning migrating to gRPC later this year.

Cadence can potentially run on any database that supports single shard multi- row transactions as a backend. Currently it supports Cassandra and MySQL.

SWF has pretty tight throttling limits. Cadence scales very well with use cases in production that require 100s of millions of open workflows and thousands of events per second.

SWF has pretty tight limits on individual payloads and number of events. For example maximum activity input size is 32k. Cadence currently has 256k limit. SWF history size limit is 10k events while Cadence limit 200k. All other limits are also higher.

Cadence has no limit on the activity and workflow execution duration.

Cadence through archival supports unlimited retention after a workflow closure.

SWF has Java and Ruby client libraries. Cadence has Java and Go client libraries.

SWF Java library is fully asynchronous and relies on both code generation (through annotation processor) and AspectJ. It is hard to set up, doesn't play well with IDEs and has very steep learning curve. Cadence Java library (as well as Go one) allow writing workflows as synchronous programs which greatly simplifies the programming model. It also just a library without any need for code generation or AspectJ or similar intrusive technologies.

Cadence client side libraries have much better unit testing support. For example the Java library utilizes an in-memory implementation of the Cadence service.

Cadence features that SWF doesn't have:

Workflow stickiness. SWF replays the whole workflow history on every decision. Which means that a workflow resource usage is proportional to O(n*n) of number of events in the history. Cadence caches workflows on a worker and delivers only new events to them. The whole history is replayed only when a worker goes down or the workflow gets out of cache. So Cadence workflow resource usage is O(n) of number of events in the history. For large workflows it makes a huge difference. It also leads to higher per workflow scale. For example it is not recommended to have workflows that execute over a hundred activities in SWF. Cadence routinely executes workflows that have over thousand activities or child workflows.

Query workflow execution. It allows synchronously get any information out of a workflow. An example of a built-in query is a stack trace of a running workflow.

Cross region (in AWS terminology) replication. SWF in each region is fully independent and if the regional SWF is down all workflows in the region are stuck. Cadence supports asynchronous replication across regions. So even in the event of a complete loss of a region the workflows continue execution without interruption.

Server side retry is an ability to retry an activity or a workflow according to an exponential retry policy without growing the history size.

Reset is an ability to restart a workflow from any point of its execution by creating a new run and copying a part of the history. For example the reset is used to automatically roll back workflows to the point before a bad deployment that was rolled back.

Cron is an ability to schedule a periodic workflow execution by passing cron string to the start method.

Local activity is a short activity that is executed in the context of a decision. It uses 6x less DB operations that a normal activity execution.

Long poll on history allows to efficiently watch for new history events and is also used for efficiently waiting for a workflow completion.

Cadence uses the elastic search for visibility. Soon it is going to support complex searches across multiple customer defined columns which is far superior to the tag based search SWF supports.

If decider constantly fails during a decision SWF records a few events on every failure eventually growing the history beyond the limit and terminating a workflow. Cadence supports transient decision feature that doesn't grow history on such failures. It allows continuing workflows without a problem after the fix to the workflow code is deployed.

Cadence provides command line interface

Cadence Web is open sourced and is much nicer than the SWF console.

Cadence supports local development through unit testing as well as using local docker container that contains the full implementation of the Cadence service and the UI.

Cadence doesn't yet have activity and workflow type registration. The advantage is that changes to activity or workflow scheduling options do not require version bumps that affect clients.

<comment><USER2> Wow thanks for the completeness of the answer.

I found myself nodding on all the conceptual limits and features you added, I'm sold, gonna try Cadence asap :)

The tooling around SWF (web console and ability to get insights about tasks, failures, etc.) is definitely a big one from an operational perspective. The SWF console is indeed absolutely terrible (with basic bugs not fixed for years, like broken pagination), so we ended up developing our own here at Botify, along with a python based client lib that mimics most of RubyFlow principles. I'm curious if all this can be integrated with Cadence, will have a look. I can keep you informed if you feel it's valuable for the Cadence project.

<comment><USER3> Besides the UI Cadence provides a CLI that supports most of the API features.

The core API is almost the same, so porting an existing python client should not be a very large task.

</comment></comment></comment></comment><comment><USER4> I've been working on a similar workflow engine for node at [https://www.npmjs.com/package/@node-ts/bus- workflow](https://www.npmjs.com/package/@node-ts/bus-workflow)

The main objective of workflows is to manage long running processes. By processes I mean business processes like coordinating the activities of fulfilling a customer order (settling charges, picking inventory, packing, dispatching, email receipts etc). It's a way to keep all those individual commands decoupled but coordinate them at a higher level.

This isn't a new concept by any means, and is often paired with Domain Driven Design and message based systems. Doing so gives you a library of events everytime something happens in your system that can be reacted to in a workflow.

If you've ever dealt with microservices, or even a monolith where two internal services are incorrectly coupled together then this approach may be worth looking into.

<comment><USER5> Thank you. I feel kind of silly about this but I feel like I've had a hard time understanding when an org should, or could use something like this. I have seen them mentioned but every time it's explained it's explained with more abstract language on top of it that confuses me. I keep hearing "it manages business processes" but then it fails to mention if this means like, a human being's process within an org, or something coupled with an application of some sort that has business processes in the application? Does this type of thing replace sort of what Jira does, make a ticket and then pass it off to the next team or whatever? Do you ship it with the app for on-premise deployments of a software product? I have a hard time seeing the big picture with things like this sometimes. Then I hear workflow orchestrator and I think, oh okay so like ansible, but for, work...flows? But what is a workflow really exactly?

<comment><USER6> This could also be used to kill off systems like SharePoint in many businesses and that would be great.

Seriously, its workflow engine has race conditions, randomly fails and has no transaction management. But there are few alternatives. I don't know why there hasn't been any real contender. You would need a full suite to challange it though.

<comment><USER7> Speaking as someone who has just implemented a complex business multi-step business process workflow in Sharepoint 2016 - I concur.

</comment></comment></comment></comment><comment><USER8> Sounds cool, but even after poking through the repos I still don't fully understand what it does.

<comment><USER9> The linked talk provides context and use cases within the first 10 minutes or so: [https://atscaleconference.com/videos/cadence-microservice- ar...](https://atscaleconference.com/videos/cadence-microservice-architecture- beyond-requestreply/)

</comment></comment><comment><USER10> Looking at a few comments, can someone answer on whats are the nuances between managing data workflows vs service workflows ?

</comment><comment><USER11> Does anyone know what are the best OSS Orchestration Engines? Am wondering what I should be comparing this to.

<comment><USER12> I don't know about best, as there are lots of trade offs between various projects. I've reviewed and worked with a lot of Java based ones, top of mind:

\- Airflow

\- jBPM

\- Enhydra Shark

\- Activiti

\- Netflix Conductor

Most of those are more workflow engine than pure orchestration.

A few years back, we searched high and low and I was generally unhappy with the commercial offerings so we wound up building around Netflix Conductor (after a disastrous run with Joget, which is built on Enhydra Shark).

Since then I've been pretty happy with Conductor, submitted numerous PRs and accidentally became one of the people keeping the MySQL backend implementation going forward.

<comment><USER3> AFAIK Netflix Conductor was inspired by the AWS Simple Workflow engine. The Cadence is a direct evolution of SWF in the open source world. I was tech lead for both of them :).

</comment></comment></comment><comment><USER13> Workflow engine = can support hundreds of parallel workflows. This includes airflow, luigi, dagster, appworx, are used to manage data and are typically processes that run in minutes to hours. Orchestration engine = can support millions of parallel workflows. This includes Uber Cadence & Netflix Conductor, are used to manage services and are typically processes that run in microseconds to minutes.

<comment><USER3> Cadence does support processes that run unlimited time. We have workflows in production that are always running. For example there are services at Uber that have an always open Cadence workflow per rider.

<comment><USER13> Very interesting. Can you give an example where you'd want to track a process that takes days or months to execute?

<comment><USER3> For example the Uber loyalty program needs to accumulate points (similar to airline points). A customer workflow receives trip completion events and updates the state accordingly. When a certain number of points is reached some actions (mostly calls to downstream services) are executed.

<comment><USER14> For this use case, could it be solved with an asynchronous task system such as Celery? What advantages does Cadence offer for something like this, which seems to be repurposing the scheduling system of Cadence to process instantaneous events?

Also how does it listen for events? From polling?

<comment><USER3> It could be solved by Celery, but it would also require a database and Celery doesn't scale that well unless runs on top of Redis which is not really fault tolerant. Also actions invocation with guarantees and exponential retries is not trivial with Celery.

Actually at Uber large number of services are being migrated from Python/Celery to Go/Cadence.

</comment></comment></comment></comment></comment></comment><comment><USER15> I thought they used Piper (based on Airflow) for workflows at Uber. [https://eng.uber.com/managing-data-workflows-at- scale/](https://eng.uber.com/managing-data-workflows-at-scale/)

<comment><USER16> That is for data workflows (your etl jobs basically). This is for services.

<comment><USER17> If you already have a service that requires ETL, Cadence might be a good choice as well. You will need more boilerplate and setup compared to Airflow/Piper, but you can share data structures between ETLs and services that use it and have more control over execution and deployment as you own your workers.

</comment></comment></comment><comment><USER18> Hi I watched the Cadence talks and read through the golang code a while back and love what you're doing with cadence. Really glad to see you're moving from Thrift/TChannel to protobuf/grpc - that was a blocker before.

If anyone could help me understand the following I'd appreciate it...

I understand that the event history is cached at worker nodes and the whole history of events is only delivered to workers if needed (ie if out of cache) and that normally cadence manages to deliver events for the same workflow to the same worker.

My question relates to what exactly happens within a _single_ worker process.

On each new event, does the worker process loop through all events in the history - starting from the beginning - in order to get back its internal state, and _then_ process the new event, and then shutdown ready to repeat this for the next event.

Or... does cadence keep the internal state around by keeping the goroutine alive waiting on a channel midway through it's workflow logic and waiting for the next event to continue execution.

Thanks

<comment><USER3> It is the latter. The workflow state object is chached including the goroutines the workflow code is blocked on. And the new events are applied to the cached object.

</comment></comment><comment><USER19> If you want to do your workflow in your own procedural code, SWF (and presumably Cadence) are good choices. If you want to use a dependency graph, Airflow is for you (but I hear operating it is kind of tricky). If you like a state-machine/flowchart kind of approach, AWS Step Functions.

AWS customers these days seem to mostly like Step Functions, although SWF isn't going away, and lots of EC2 instances are running Airflow. Obviously, some people want a managed service and others want OSS that they can control & fine-tune. Nothing wrong with either choice.

Most of the engineering cycles these days are going into Step Functions, keep an eye on that space.

<comment><USER19> (Oh, should disclose, I helped design & build Step Functions.)

<comment><USER3> If only SWF was extended to run deciders on AWS Lamda. Without this the main advantage of the Step Functions is hosting. I personally would rather see an integrated system where the Step Functions are a natural extension of SWF not a completely separate system.

This is the direction the Cadence is going. We are planning to add support for integrating custom DSLs easily, but maintaining the core code based libraries.

BTW: If anyone is interested on running the Step Functions DSL on top of Cadence contact the Cadence team. We could work together to get it implemented.

</comment></comment></comment><comment><USER20> Hello, I am very interested in knowing the open source version of SWF. I am heavy SWF user and I watched all Cadence videos especially the architecture one. I wonder what is the shard recovery mechanism for a single shard on one host ? Let's say one host is down, how the shards on that host are recovered on the next host ? I heard the presenter said consistent hashing or RingPop, so I am thinking all shards should be migrated to the next available host or hosts ?

</comment><comment><USER21> How does this compare to netflix conductor? I've just started experimenting with it and somehow I missed this during my evaluation

<comment><USER3> I'm from Cadence team, so I'm obviously biased :).

Besides very different implementation backends the main difference is that Conductor defines workflows through a JSON based DSL and Cadence defines workflows as code. Because of that it is possible to extend Cadence to interpret Conductor DSL, but the reverse is not possible.

I believe that any non trivial workflows that have some state management requirements are more easily expressed as code. Any attempt to come up with JSON or XML or YAML or whatever language for workflows is always be inferior to existing programming languages like Go, Python or Java.

<comment><USER22> How does this work ? Do you store the code for the workflow in the database ?

<comment><USER3> Cadence is a service. The workflow and activity code lives outside of it. Think about Cadence workflow and activity code the same way you think about a queue consumer which is external to the queueing service.

<comment><USER22> No - I'm wondering about the abstractions that allow for specific workflow specifications to happen in code versus a DSL/JSON.

There are two extremes here - I can slap celery and run a bunch of custom code as workflows. On the other hand, I can use a workflow system with a built in DSL that abstracts some of the underlying behaviour out.

Cadence seems to fall in the middle - and I'm wondering how it works. Why doesn't it generate to the same mess that celery+a bunch of custom python code become ?

<comment><USER3> Cadence is above :).

It allows:

Integrate any DSL without modiying the core service. Internally at Uber there are at least half dozen DSLs running on top of it.

It allows to write code that hides all the complexity that leads to the mess of queue + db implementations. The beginning of this talk explains the idea: [https://youtu.be/BJwFxqdSx4Y](https://youtu.be/BJwFxqdSx4Y)

The gist of it is that you write just your business logic without thinking about callbacks and storage.

I recommend looking at the Cadence samples to get the taste of it. Join the Cadence Slack channel if you have any specific questions: [https://join.slack.com/t/uber- cadence/shared_invite/enQtNDcz...](https://join.slack.com/t/uber- cadence/shared_invite/enQtNDczNTgxMjYxNDEzLTI5Yzc5ODYwMjg1ZmI3NmRmMTU1MjQ0YzQyZDc5NzMwMmM0NjkzNDE5MmM0NzU5YTlhMmI4NzIzMDhiNzFjMDM)

<comment><USER22> Thanks for a detailed reply. Really appreciate it. We have an internal workflow engine that's nice, but still has numerous quirks. We are trying to learn better.

I'm wondering why you didn't create a DSL in the first place...if it's ending up in DSL all over the place. To choose an example - why did you go the kubernetes way with yaml . Sure it's verbose as hell...but there aren't multiple forms of yaml that could bitrot.

Why do language primitives matter ?

Let me also ask you another related question - suppose you wanted to build a GUI that builds workflows for your business teams..are you saying you generate language code ? Or do you generate yaml/JSON and then interpret it using your worker code ? Again - the same question: wouldn't it have been better to have a uniform declarative JSON ?

<comment><USER3> DSL is DOMAIN specific language. But it is common mistake to call GENERIC workflow definition a DSL.

My opinion is that DOMAIN specific languages are awesome when they are used for a specific narrow domain. For example AWS CloudFormation template is a DSL for a cloud deployment. If This Then That, also known as IFTTT is another good example of a narrow workflow definition.

At the same time a generic turning complete language in JSON/YAML/XML always starts simple but ends up as a complete mess. See [https://mikehadlow.blogspot.com/2012/05/configuration- comple...](https://mikehadlow.blogspot.com/2012/05/configuration-complexity- clock.html). Any programming language is much better for writing complex programs. The problem is that most existing workflow/orchestration systems force developers to use unnatural programming patterns and libraries to make the code fault tolerant. Cadence is an attempt (pioneered by my team at AWS Simple Workflow and later picked up by Azure Durable Functions) to implement workflows as natural programs without much boilerplate concepts and code.

Think, why nobody tries to write complex backend programs in JSON? The reason that programming languages have well defined ways to deal with complexity. JSON based languages are good for limited domains, but anything complex makes them unusable. I've seen it hundreds of times already when DSL is abused and developers hate it.

So most of the Cadence workflows are written directly as Go/Java code. But when DSL is appropriate it can be added trivially by interpreting it by the worker code.

>Again - the same question: wouldn't it have been better to have a uniform declarative JSON ?

Again, declarative JSON is declarative only in narrow domains. The generic workflow definition language in JSON works only for very simple scenarios and is harder to write and debug then Go/Java code.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> Is this like Celery + RabbitMQ but with a GUI? On a high-level it sounds like that, but could someone be so kind as to give an example use case of this? How is it different that it's not just described as a distributed task queue?

<comment><USER3> The main difference is that workflow has state and tasks can be very long running. Also workflow can react to external asynchronous events. Visibility into overall progress is also a very important feature. When using a queueing system it is hard to answer about the current state of the business process.

For example implementing service deployment to a public cloud usign Celery + RabbitMQ is very non trivial and error prone. It is a pretty streightforward Cadence workflow.

</comment></comment><comment><USER23> Cadence is also the name of an electronics design automation company.

[https://www.cadence.com/](https://www.cadence.com/)

</comment><comment><USER24> What does the workflow versioning story look like? This is one of the most frustrating parts of SWF that Step Functions + Lambda has effectively solved for us.

<comment><USER3> Cadence doesn't require workflow and activity type registrations which eliminates most of the problems with SWF versioning. Cadence supports versioning of the workflow code out of the box. Any change has to be protected with a version condition. It works even with shared libraries and very long running workflows.

</comment></comment><comment><USER25> Can somebody from Uber clarify? Uber could've used Cadence instead of Piper / Airflow isn't it. Please correct me if I'm wrong?

<comment><USER3> Yes, it is technically possible to use Cadence instead of Piper/Airflow. The reason the Piper/Airflow is used is mostly historical. It was around for quite a time and Cadence is a relatively new project.

</comment></comment><comment><USER26> How does it compare to BPM solution like Camunda?

<comment><USER3> Cadence allows to write workflows as code. Think of it as a virtual machine for OO code that makes that code fully fault tolerant to process failures. As it is code it can be used to implement any business logic. Camunda is BPMN engine which interprets BPMN workflow definition. It is possible to implement a Cadence workflow that interprets BPMN without changing the core Cadence service.

<comment><USER27> And we are talking about a software company and a software package. Sounds like a strong overlap to me - or do you think you can call a new software library "microsoft"?

</comment></comment></comment><comment><USER28> how does this compare to Luigi or Airflow?

<comment><USER3> The biggest difference besides the programming model is scale. Luigi and Airflow target data pipelines that don't require much scale. Cadence is built to support business level transactions. It can handle tens of thousands of events per second and hundreds of millions of open workflows. Obviously it is also a good fit for low scale use cases. Cadence is also so generic that it can be used to implement practically any workflow definition language. For example it is possible to create an extension to run Airflow pipelines on Cadence.

I'm from the Cadence team.

<comment><USER29> > Obviously it is also a good fit for low scale use cases.

Is this a given? Just because something is necessary at scale doesn't mean it is a good fit for low scale use cases. I would expect the opposite is actually true.

<comment><USER3> We have both very low scale use cases as a single distributed cron as well as very high scale use cases in production at Uber.

</comment></comment></comment></comment><comment><USER30> just a random comment: I thought Cadence, the EDA company, owns the trademark for the name

<comment><USER31> Trademarks are partitioned by subject matter: [https://www.uspto.gov/web/patents/classification/selectnumwi...](https://www.uspto.gov/web/patents/classification/selectnumwithtitle.htm)

</comment></comment><comment><USER32> How does this compare to Argo?

<comment><USER3> Argo workflows are DAGs written in YAML. The types of workfows you can create using this syntax is very limited. Cadence gives you full power of a programming language like Java or Go to implement workflow logic. It is possible to implement support for argo DSL on top of Cadence. The reverse is not possible.

Cadence is also more scalable supporting tens of thousands events per second and hundreds of millions of open workflows.

</comment></comment><comment><USER33> Does Cadence work for low latency scenarios? Ex. web serving?

<comment><USER34> We use Cadence to run low-latency workflows for routing customer support tickets. It handles our use case very well. [https://eng.uber.com/customer- obsession-ticket-routing-workf...](https://eng.uber.com/customer-obsession- ticket-routing-workflow-and-orchestration-engine/)

</comment></comment><comment><USER18> The way I think about workflow engines is as follows. Please comment and correct me. Keen to discuss.

Workflow engines like Cadence essentially work by letting you write regular- looking procedural logic for your workflow. This looks and feels very much like writing an async function with async-await in javascript or C#.

The state in your workflow is then implicit in your code instead of explicit. Here's what I mean by that:

Usually you would _explicity_ serialise your state between each incoming event and do an atomic-compare-and-set operation on an external database to store the new state.

For a new incoming event: (1) fetch the state, (2) marshal into an object in your programming language (ie a java class or golang struct), (3) given the current state, process the event and perform any external actions like sending an email. These external events need to be ok with being done with at-least- once semantics. Update the state object ready for the next incoming event. (4) serialise the state, (5) store the state in the database (atomically update with a compare-and-set operation). (6) Repeat on each event. Do everything with at-least-once repeat-on-failure semantics.

In a workflow engine like cadence, what is persisted to the database is the entire history of events instead of a single state object as described above.

In Cadence the code you write looks very much like async-await style code in languages like javascript or C#. The workflow logic is in some sense an async function that pauses at await statements and picks up again where it paused when the next event comes in.

Remember that cadence stores the entire history of events for a workflow. It does this so that it can rerun the workflow from the beginning, this time with the new incoming event on the end of the history.

Notice that you need to be careful about your workflow being deterministic.

Optimisations: (1) it knows when it is replaying already-seen-events and doesn't redo external events such as sending emails. (2) it tries to resend events to the same worker node each time. It caches events at worker nodes. (3) at the macro level everything is highly-available and repeat-on-failure- with-backoff to ensure progress and at-least-once-semantics. (4) it supports repeating workflows and child workflows (5) monitoring, tracing, other things you'd expect (6) etc

Importantly: notice that there is still in some sense a single state object. The state is just implicit: it is deep down in the internal state machine of the language you wrote the workflow function in (java, golang). Instead of serialising state such as 'time-since-last-email' in a state object to a database... you have 'time-since-last-email' as a local variable in the scope of the workflow function. Similarly your programming language is tracking the call stack and current execution position of the function... normally you'd keep track of progress through the workflow in the state object and condition on this state when receiving a new incoming event.

Thinking about state as explicit (state object approach) versus implicit (replay-history approach) helps me when thinking about cadence and similar workflow engines.

....................................................... Thanks for reading so far. I'd love to hear from users of cadence at uber or elsewhere:

(1) why do you choose to write workflows with implicit state (by replaying history) instead of storing the state explicitly as a serialised state object in the database?

My guess: developer productivity of writing and maintaining the workflows. Having a common approach and single observable system for many different workflows.

(2) how do you reason about long-running workflows where the business logic needs to be updated? Would this not be much easier if the state object (say a serialised protobuf) was stored explicitly in the database?

(3) wouldn't non-determinism be much easier as well if you stored the state explicitly?

<comment><USER3> (1) It simplifies the programming model. There is no way to serialize a state of the call stack through a library in most programming languages. You mentioned that Cadence is similar to C# wait/async. The SWF Flow library is. But the Cadence workflow code is fully synchronous, not requiring callbacks unless needed by the business logic. Applying new events to the cached workfow is also more efficient for large states.

(2) It depends. Nothing prevents a workflow writer from checkpointing the state explicitly (by calling continue as new). Infinitely running workflows do it periodically. But having the events history is awesome for rollbacks. For example in Cadence it is possible to rollback a bad change and automatically rollback the state of all your workflows to the good state. In database world a change that corrupted the state is much harder to deal with.

(3) The experience shows that the determinism requirement while requires some learning is not that hard to deal with. But the superior programming model it allows is liked by users.</comment></comment>


Ask HN: How do I transition from full-stack web-dev to systems engineering? - <AUTHOR> I've been working for the past year as a software engineer. Half of that was for a consultancy, and half as a backend engineer writing business logic / REST APIs, + some moonlighting for friends and family who need a site or app or data munging help. Before writing code professionally, I worked for about 10 years in product-adjacent and operations roles.
I'm self taught. I love to code and spend most of my free working on personal projects and trying to fill in the gaps in my CS knowledge.
That said, I'm finding myself much more drawn to creating tools and languages than working on CRUD/business applications. It's a dream of mine to work on building runtimes and server infrastructure rather than the business software that sits on top of it.
I've been learning C and Rust and PL implementation, working through operating systems books, and recently picking up assembly from that and GameBoy programming. That said, I feel like the amount I need to learn is crushing, and I'll still have no idea what it's like to apply that knowledge in a production setting. The job postings for systems engineers I've been seeing all seem to require several years of experience + experience managing systems with thousands of hits per second + contributions to the Linux kernel + a degree. I can't go back to school at this point in my life and barely have time for the side projects and self studying that I do. So, feeling a bit defeated.
Questions/tldr: 1. Is it realistic for someone to become a productive self-taught systems engineer? Are you or someone you know a self-taught systems engineer? 2. I get that systems engineering probably looks different at different companies, but if you are a systems engineer, what concepts do you find yourself calling upon on a daily basis to do the work that you do? I'm asking this with an eye to paring down the different subjects I feel I need to master to get where I would like to be. 3. If you were in my shoes, how would you proceed? <comment><USER2> The term, systems engineer, had different interpretations depending upon the environment and context. In my experience, in IT the term could be better described as systems integration, i.e. building systems out of sub-systems. I have engineering degrees and decades of experience and yet I wouldn't claim to know more than a fraction of what you allude to in your question.

Very loosely speaking there are there major areas of systems engineering, mainframe based, large-scale Windows enterprise server based and Linux cloud based. The technologies differ greatly across these so you probably should choose a single area.

The common ground amongst these three areas is network engineering. Any large scale environment comprises complex interconnection of nodes, switches, load- balancers, etc.

If I were in your shoes, I would find a job with a medium sized systems integrations consultancy and then develop my skills in one area at a time. Over time you would end up with a comprehensive knowledge across the many areas.

</comment><comment><USER3> fwiw, You might enjoy what golang has to offer in a systems programming context. I think it's reliably offered it's users '95% right, 99% of the time' comprehensiveness for all kinds of contexts -- what with how I think even Dropbox might have switched over to Golang, from Rust, after conceding it's greater overall compatibility (I'd imagine) to their product's improvement + maintenance + growth/scale.

<comment><USER4> Dropbox is using both Go and Rust; they've haven't switched away from either. They even use both together in some places.

Where did you get that idea?

</comment></comment><comment><USER5> I too started w/ web dev (PHP and HTML (so funny to think about those first days starting out)), but ended up moving to the backend (Apache Suite) for my current job, and took up lower level programming from there, so here's my 2 cents.

#1 - throw out the "systems engineer" title. It's ambiguous, can't really be _achieved_ , and I've seen guys out there who are actually Systems Engineers (in job title) who are so good that it's almost sickening - makes me want to quit. So first, get rid of the title.

#2 - Learn a low-level language. You seem to be already doing this (I'm a RUST guy myself). Here you'll probably see (if you haven't already), that there's a focus/efficiency that isn't _necessary_ for CRUD apps (e.g., passing references, using as little memory/cpu as possible, etc). But that's kind of what I like about lower languages, you get almost intimately close to the machine and understand on a completely different level that a web dev.

#3 - And this is something I'm still dealing with: switch your dev mind from higher-level web work, to lower level work. What do I mean? Well a lot of times in web dev, you're building at the application level (e.g., CRUD apps). However, I'm noticing for a lot of lower-level programming, you're building _beneath_ the application level. I'm working on a load balancer at the moment, which isn't really a part of the _application_ , but kinda sits underneath it. The thing about lower-level dev work is that it's not really the show-boaty type of thing that a React can be (oh look at my shiny UI). And in my opinion, it doesn't get celebrated like it should have. I have SO much more respect for lower-level guys, after jumping into that world.

TLDR; systems engineer title is ambiguous, learn a solid low-level language and stick with it, reframe your thinking when it comes to designing/planning what you build</comment>


The first intuitive programming language for quantum computers - <AUTHOR> https://ethz.ch/en/news-and-events/eth-news/news/2020/06/the-first-intuitive-programming-language-for-quantum-computers.html <comment><USER2> Now I just need to build a quantum qBit HTTP server, take some DMT, interface with CERNs new super-collider and I've completed the game of life.

</comment><comment><USER3> Some more info here:

[https://silq.ethz.ch/](https://silq.ethz.ch/)

<comment><USER4> Direct link to the referenced paper in the article [PDF]: [https://dl.acm.org/doi/pdf/10.1145/3385412.3386007](https://dl.acm.org/doi/pdf/10.1145/3385412.3386007)

</comment></comment><comment><USER5> According to George Hotz, we need to be careful how many computations we throw at a quantum computer, due to it making our simulation too expensive and cranking someone's hosting bill.

</comment><comment><USER6> I have a feeling that the right solution isn't a new programming language, but a library with a good API in an existing and proven language. Maybe. A quantum chip is a weird compilation target.

</comment><comment><USER7> Looks interesting but a long way from my wheelhouse.

I'd have been tempted to name it Wirth though given the source institution.</comment>


Be a nurse or P.A. instead of a doctor: the underrated perils of medical school - <AUTHOR> http://jseliger.wordpress.com/2012/10/20/why-you-should-become-a-nurse-or-physicians-assistant-instead-of-a-doctor-the-underrated-perils-of-medical-school/ <comment><USER2> First, I know the medical field quite well, as I'm working in it, and I can emphatically say that nursing is quite saturated now. It has always gone through boom and busts, and currently, it's going through a bust. Just read the All Nurses forums to see all these threads about new grads being unemployed and unable to find jobs for a year or more. It's definitely tough out there and will only get tougher since SO much nursing programs have become accredited in the past decade.

Secondly, being a nurse is a HARD job. It's a blue collar job, and it should be (probably was) on Mike Rowe's Dirty Jobs. It's back breaking labor, stressful, and debilitating. Most of my family work in health care, and so far, my cousin has three herniated discs from lifting obese patients over the years, my wife had several blood tests done from being exposed to Hep C patients, and I've known several nurses who have suffered extreme burn out.

Being a PA is marginally better, I suppose, but you're dealing with the same garbage.

There are some definite pros about becoming a nurse over a doctor for sure. But, if you plan on staying on working the floor, become a doctor if you're smart enough. It's still in nearly every regard better than being a nurse on the floor. However, if you have more ambition, you can try to move up the ladder in management and administration as a nurse.

Finally, there are lots of other oppty's for nursing besides beside nursing. If one gets burnt out on patient care, one could move into case management, school nursing, teaching, etc...

<comment><USER3> I second this. My wife is a biologist by training but went to nursing school (this was back in 2000) to ensure she'd always have a job, in any economy. She fell in with a nationally known urologist MD/PhD during the program and got an offer before graduation to start working in his clinic attached to the university. The pay was awful (~$35k/yr at that time) because of the glut of fresh nurses minted by the uni each year. The work wasn't bad in a clinic -- standard office hours and no obligation to pick up shifts at the hospital -- and it was a great opportunity to get involved in real research (she published in her first year there), but it was mostly repetitive and boring, and again, the pay stunk. When my company relocated me out of state, she switched over to drug safety & pharmacovigilance at a CRO, and then at a pharma. Most of the roles in those areas require a health care provider background (either RN/BSN or MD/RPh) and the stress & money are muuuuuuuuuuuuch better. I was a senior IT manager with a team of about 80 people in 10 countries and she was making more than I was (back in about 2009).

Nursing school is much easier than medical school, and there are still perks to having that professional credential, and, as she and her colleagues always joke, they can "always go back to the hospital if things get bad". That said, my next door neighbors are both consulting radiologists and work from home. I barely see them, but by all appearances they're making bank while exerting hardly any effort. My childhood friends who've become doctors are a pediatrician and an ER doc, respectively, and they work much harder but they love the career they've chosen. To each his own: in every field there are ample opportunities for the top performers.

</comment></comment><comment><USER4> Don't be a nurse, be a dentist. 4 years of schooling beyond BS, no residency, $146,920 per year median (BLS, 2010). In several states "dental culture" is to only work 4 days a week.

In cities around the size of Raleigh, NC a practice with one experienced dentist can generate 800k-1.2m in revenue and end up with a 250k-300k take home. Of course it takes a few years to build up the patient base. Starting salaries for dentist in Raleigh working for a larger practice is 150k-170k. (Source: My wife is a fourth year dental student).

<comment><USER5> What is the barrier to entry? i.e. What makes this hard? Is it just "being the bad guy"?

edit: not to sound like a jerk, but I'm curious

<comment><USER6> Four years of extra school is a pretty big barrier.

<comment><USER7> So is getting a PhD. Seems like being a dentist is a better alternative in monetary terms than being an English major student + PhD, sociologist with a PhD or even an engineer with a PhD.

<comment><USER6> Oh, there's no question that's true. Most PhDs aren't even close to being worth the money. Getting a tenured position these days is like hitting the jackpot.

</comment></comment></comment></comment></comment><comment><USER8> French person here, a lot of my French friends from high school went into med school. Free universities, no debts, heavy unionization, you can easily move to nursing/pharmacology if you fail med school... free education solves many social problems.

<comment><USER6> It's not really free, though. It's just paid by someone else.

<comment><USER8> It's free for the students; that's one of the points of the article, that financial debt can be an obstacle to personal liberties.

<comment><USER6> High taxes are an obstacle to personal liberties, too.

<comment><USER9> So are children and stepmothers. Your point ?

> High taxes are an obstacle to personal liberties, too.

No. Taxes are spread out among the population. Financial debt is burdening only one individual.

In Europe, in France and Belgium, we like to think solidarity is a good thing.

<comment><USER10> That's what's wrong with a lot of Americans, IMO. It's every man for himself, screw your neighbors. If something bad happens to someone else, they deserved it for reason X. (X being: they didn't work "hard enough," they were "lazy," etc when the person in question is often less lazy and harder working than the person complaining about them)

I actually had a classmate tell me the other day that me being able to get the medications I need to stay alive (because of the ACA) was going to drive the country bankrupt and (direct quote here) "destroy America." In my experience, my countrymen do not care for others very much--only themselves.

</comment></comment></comment></comment></comment></comment><comment><USER11> That is the most harmful advice you could ever give to someone.

Who do a startup, if you can do a PhD, it's not that strenuous. Why do a PhD if you can just do a Masters, it's not that strenuous. Why go to university, if you can just go to a community college, it's not that strenuous. Why finish high school, if you can just do an honest labor job, it's not that strenuous.

See the false assumption here? 5 years from now, these people will regret that they didn't start 5 years ago. Hard work is builds you as a person, it makes life more interesting, it gives you purpose and after these 5 years, you can decide your own pace. You have the OPTION.

Not going for higher education, limits your options FOREVER. You can't change it anymore, it's final. If you're not looking for meaning in life, go for the lesser option, if you are looking for meaning, go for the highest option you can think of and then even one more. The funny thing is, once you're getting closer to your highest option, you discover that there is still more to discover and that it is beautiful.

For that reason, shoot for the stars, don't listen to nay sayers, make your life awesome you only have 1 and then it's over.

I'm still cringing, hope I could get my point across.

<comment><USER12> >Not going for higher education, limits your options FOREVER. You can't change it anymore, it's final.

I didn't go to college until I was nearly 30. They still let you in.

<comment><USER11> And that's the reason, why it is so harmful to not go for higher education. If there were only more people thinking that way, you might have heard of it and pursued it. Instead you had to throw your whole life around again at 30, where you could have been much further already.

But awesome that you took the step.

<comment><USER12> I'm saying that it wasn't such a big deal, and it didn't affect anything FOREVER. I did a lot of things before I went back to college, and I've done a lot of things after.

>could have been much further already.

I do well. If this is peak income for me, it's an excellent peak income. I'm not looking to go "further" unless random events happen to drag me there. To be clear: if I had it to do all over again, I would have waited just as long to go to college. Out of all of the things I would change, that's not one of them.

edit: also, people who go to college do not have the monopoly on hard work. As a person who did an awful lot of manual labor in my life, what I do now doesn't even feel like proper work. I feel like I'm stealing a paycheck for doing puzzles all day.

</comment></comment></comment></comment><comment><USER13> I'm currently a resident (not in primary care), so I'm biased. But:

* Nursing sucks. It's terrible work. On top of that, you have to deal with jerk doctors and jerk patients. Not a single one of my many doctor friends would ever have considered nursing over med school. (Side note: if you meet a good nurse, thank them! Truly good nurses are rare.)

* PAs generally do not know all that much about medicine. With very few exceptions, it was frankly terrifying signing out my service to them every night, and never fun cleaning up their messes in the morning. But with resident work hour rules, there is no alternative.

* The bottom tier of medical students/residents aren't very good either. Pay doctors less and the smart ones will realize that maybe there is some other way to achieve life satisfaction, because money won't be a differentiator any more. Care will get worse but the general public won't notice or care. They will keep getting fat, diabetic, and ever sicker, insist on keeping their essentially vegetative elderly relatives alive at all costs, and then complain that healthcare is expensive.

</comment><comment><USER14> One unmentioned upside to being a doctor is you make a good salary anywhere in the country. A GP can make $180K in a town where an engineer would be lucky to make $40K, if any engineering jobs existed at all. PAs are mostly in the same boat but top out around $90K. Nurses are usually in demand in most places but the pay can be lousy. Every other well paying job is very location dependent.

<comment><USER14> Also, one thing I forgot to add, is that PA school is very expensive.

For example, at the University of North Dakota, the PA school tuition is $34,287 per year, with no discount to in-state residents. In contrast, the medical school tuition is $24,998 per year for out of state students, and $12,815 for in-state students.

</comment></comment><comment><USER15> It doesn't have to be this way. Why is it this way? There's a complicated answer and a simple answer. The simple answer is because human resources management for doctors is non-existent. It's an unknown, alien concept. Example - it required a major paradigm shift just to make a rule that doctors can't work for 36 hours in a row. So the question is, do you want to put up with all this because nobody can be bothered coming up with a decent way to manage a key part of the healthcare workforce? Or more correctly, because doctors can't get their shit together to demand decent working/training conditions?

<comment><USER10> In many ways, I feel like medical culture is very old-fashioned.

Truck drivers aren't allowed to drive if they haven't gotten any sleep, but your doctor in the ER making the decisions that will either keep you alive or let you die will almost always insist that 36 hour shifts are essential. There's a lot of ego involved, as can be seen in the way that that same physician will usually support limitations on truck drivers but will readily believe in their own inability to make mistakes--even in the face of scientific research showing otherwise.

<comment><USER13> At my hospital, residents take 24-hour calls. The trouble with shorter calls (say, 12 hours) is increasing the number of transfers of care. With 25 patients on a service, spending even five minutes on each means each "shift change" would take two hours. So you spend closer to one minute on each, and at that point it's easy to miss important details. We are skeptical of the "scientific research" because we've all seen errors made because of poor signout that were not happening before work hours were so restricted.

For the record, attending ER shifts at my hospital are closer to 8 hours, and residents do 12 hours.

<comment><USER10> I'm familiar with the transfer argument, but frankly find it a little alarming. Doctors are supposed to be very smart, but they're apparently unable to write down some sort of bullet point list or short paragraph about the state of a patient for the next person coming in? Good programmers document their code. The equivalent goes for doctors, too.

It doesn't seem like you care much about it ("scientific research"), but IMO it's much more likely to put programs in place to help doctors learn basic literacy/documentation skills than to subvert the basic biological truth that staying up for as long as most medical students/residents/etc are forced to do puts them in a state equivalent to drunk driving. ([http://archives.cnn.com/2000/HEALTH/09/20/sleep.deprivation/](http://archives.cnn.com/2000/HEALTH/09/20/sleep.deprivation/)) I do NOT want someone crashing my health into a ditch because they were too tired to miss something important. Also if you don't have enough doctors? Maybe you should start training more of them, instead of artificially limiting supply. (Residency placement slots are a whole other issue, of course, but still not something that should stymie a first world country so.)

I've had way too much experience with rude, sleepy, and grumpy residents who bungled my medical care in the last two years of frequent ER and inpatient stays to have much trust in the profession as a whole. I mean, if you're going to charge me $50,000 for your crappy medical care, could you at least not snap at me? Or forget me in an observatory room for over 12 hours waiting for a scan? They "lost" me in the system somehow until a nurse wondered why I'd been sitting there so long, called radiology, and left the room to bitch them out about forgetting me. Since the doctors couldn't be bothered to tell me what they were checking for, I only found out 13 or so hours later that the scan was to rule out a carotid artery dissection. I was in extreme pain the whole time, which they did nothing to even try to alleviate. Incredible lack of empathy.

I'm glad to hear that your hospital has gone to at least trying to adhere to reasonable shift limits; it's a sign that at least some people in some hospital systems have decided that policies leading to killing or crippling patients is something that should be prevented, not encouraged. It's certainly not something I've seen in the major medical systems in my state. :(

<comment><USER13> > Good programmers document their code. The equivalent goes for doctors, too.

The information is there; there is simply too much of it to go over with the next person in the limited time available. I understand your point about commenting code - I meticulously document all my own code.

> Also if you don't have enough doctors? Maybe you should start training more > of them

The training itself requires a huge amount of time and exposure to a large variety of situations. There are only so many opportunities for learning, and only so many hospitals which both can provide an appropriately varied case load, have attendings willing to teach, and can afford it.

I'm sorry to hear you had a bad experience. I don't shout at my patients, or leave them for 12 hours without any information.

> policies leading to killing or crippling patients is something that should > be prevented

It's not clear to those of us actually doing the work that this is actually true, when we examine our own personal experience.

</comment></comment></comment></comment></comment><comment><USER16> There are some good points in this article that most anyone thinking about entering a medical profession should consider. But there's quite a bit of anecdote and cherry picking of facts as well.

Here are a few counterpoints worth considering.

* The example used by the article for salaries is for primary care doctors, one of the lowest paid (and by any account underpaid) specialties. According to the Medscape physician compensation report [1], the highest paid specialties brought in between $350K and $400K 2012. I don't think there's many PAs or nurses making that.

* Yes, medical school and residency are hard. My wife regularly put in 90+ hours during a couple years of her education as a physician. But I'm certain there are plenty of developers on HN that put in 90+ hours during certain phases of their careers. Working very hard for a couple years won't ruin your life. By the time they're practicing - 52% of physicians see patients for 40 or less hours per week.

* The article says that, "nurses start making money when they're 22, not 29", glossing over the fact that physician residents and interns do in fact get paid. Not a ton of money, but usually around $50K/yr. So really, doctors start making about the same amount as nurses by the time they're 25, and substantially more just a few years later. And yes, medical school loans are expensive, but nursing school is far from free.

* The success rate for physicians who start their own practice is very high. I've been shocked at the amount of unsecured credit that banks are willing to extend to my wife to start a practice. It's like we've stepped into a time-warp back to 2006.

* If you're interested in living someplace rural, you can, without any negative financial repercussions. In fact, you'll probably make quite a bit more money practicing medicine in a rural area than an urban one. There aren't many careers where that's true.

* If they had to do it all over again, 51% of physicians would start again in medicine (again, the Medscape compensation survey). I don't how how that compares to other careers, but it sounds fairly reasonable to me

Being a doctor certainly isn't easy, but there are some definite benefits. One of the biggest is that it's one of the few careers left where you're pretty much guaranteed to be well-compensated so long as you do well in school.

There are certainly pitfalls, and I wouldn't advocate it to anyone who's doing it solely for financial reasons. But, if you have a passion for medicine, there are still plenty of rewards to be had by being a physician.

[1] [http://www.medscape.com/features/slideshow/compensation/2013...](http://www.medscape.com/features/slideshow/compensation/2013/public)

<comment><USER15> Just about the workload, I think that working hard in medicine is quite different to working hard in most other situations. Certainly doing long shifts in a major hospital often invovles:

1\. An entirely unpredictable work load, with no mechanism to compensate for excessive demand. 2\. Dealing with problems for which you have not been properly trained. 3\. Dealing with other doctors that are also extremely busy and stressed, and therefore limited in how much they can help you. 4\. Having to work while physically and emotionally exhausted or actually genuinely sick, with no recourse to get help. 5\. Having to do all this while studying or taking exams. 6\. Not being paid fully for the hours you work. 7\. Sometimes being sworn at, racially vilified, spat at, threatened or attacked. Depending on which job you do, this could come from your colleagues, nursing staff, admin or the patients.

I think that certain other professions such as the police or military personnel encounter similar situations, but I would hazard that they are better trained, better supported and rostered better than doctors ever are. So although many other people might work 90+ weeks, I doubt it is under these conditions. Because that would be a dumb thing to do right? And only doctors with their misguided sense of duty and high pain tolerance are willing to front up again and again. Nurses can suffer the same problems, but in general they are much better at protecting themselves.

</comment></comment><comment><USER17> I don't agree with this -- I've never been a doctor, but worked in a hospital for a few years with doctors.

The hand waving to say "you will be a primary care physician" is the worst part. Maybe you don't get our pick of specialty, but assuming you will be a pcp against your will seems silly, too, since you'd know going in roughly how good you are, and a lot of pcp are because they wanted to be.

It ignores one of the easiest/best ways to become a doctor -- the military. You give up a bit of your early career salary in exchange for a different/more interesting lifestyle for your early 30s. Still great pay.

If I were picking a career as a smart 18 year old, md/phd to go into research, or something amazing in clinical practice like a trauma surgeon or neuro radiologist, would be high on my list, independent of income issues.

</comment><comment><USER18> Disclosure - I'm an actual medical doctor.

* I agree with the overall theme of the article. That is, a lot of people go into medicine thinking that they will be "rich". And when they come out at the other end of the tunnel, they realize that the promises of job security and a comfortable lifestyle aren't really true.

Many of my classmates from medical school went into all of this with the false impression that they will make a lot of money. This misperception is because they see physicians from decades ago do really well, living in nice houses in expensive neighbourhoods, driving nice cars, etc. But unfortunately that is because 20-30 years ago, doctors' salaries were much higher relative to inflation. If a doctor made 200k in the 80's, the purchasing power "back then" was enormous. Not so today.

* The article does miss the point, as atourgates mentioned in the comments, that being a doctor is "one of the few careers left where you're pretty much guaranteed to be well-compensated so long as you do well in school."

I know a few guys who -- through brute effort -- bills upwards of 500-800k per year. Now this is gross billing, not net take home pay, but that is impressive nonetheless. It is, without a doubt, very hard work. You would have to work pretty much 6 to 7 days a week, and really convince yourself that you like money to make it all worthwhile.

* The last thing that the article barely touches upon which is important is how tough the job can be when dealing with (some) patients. Nowadays, the job is 90% customer service, and 10% medicine. Maybe less medicine -- certainly some days feel like that.

All this comes with the job's territory, but day-in and day-out of this can cause a lot of psychological stress/burden/fatigue for the physician himself.

This kind of job is not for everyone. Quite literally you have patients upset at you because they think the medication you prescribed for them doesn't work, and in the end it turns out they were not adherent on the medication (ie. they didn't use the medication or they didn't use it properly). It always reminds me how close a certain scene from House MD ([http://www.youtube.com/watch?v=dMAS2S51bM8](http://www.youtube.com/watch?v=dMAS2S51bM8)) approximates real life.

* Would I want my kids to go into this kind of a job? Only if they realize what it involves. I really do think some of the happiest jobs out there are ones where you get to be creative solving different problems every day.

Although I'm happy I'm making an impact on other people's lives, it involves very little creativity. A long time ago a specialist told me that "medicine makes you dumb." Medicine today is algorithmic: if this, then check this, then do that.

I don't think there is a simple answer whether medicine is "worth it." It is different for everybody, and it really boils down to what your personality is.</comment>


[The Record Industry] makes Enron look like amateur hour.. - <AUTHOR> http://observer.guardian.co.uk/omm/story/0,,2241544,00.html

<comment><USER2> A long but excellent piece on the history of major record labels which helped me better understand the music situation today.</comment>


LibrePCB 0.1.2 Released - <AUTHOR> https://librepcb.org/blog/2019-07-14_release_0.1.2/ <comment><AUTHOR> From [https://librepcb.org/](https://librepcb.org/)

> LibrePCB is a free EDA software to develop printed circuit boards.</comment>


These skills make you most employable. Coding isn't one – can that be right? - <AUTHOR> https://80000hours.org/articles/skills-most/ <comment><USER2> Programming appears just above Mathematics and Science ...</comment>


The Strange Persistence of First Languages (2015) - <AUTHOR> http://nautil.us/issue/30/identity/the-strange-persistence-of-first-languages <comment><USER2> In the late 90s I was on a US project which included a small Dutch software company. The founder told me about Dutch people who had been living in the US for thirty plus years, speaking perfect English, now in nursing homes, who with age or dementia or whatever, can lose their ability to speak English. They revert to their native language only. I've never heard of this as an issue again, but I wondered how this might turn into a serious problem caring for many immigrant people who lose the ability to speak English in their later years.

<comment><USER3> Same thing has happened in Sweden. There's a cohort of elderly people who arrived to the country as children during WW2, e.g., from Finland. Many were under 10 when they arrived, and now that they have gotten old, some of them have lost most of their ability to speak Swedish, and are only able to speak in their native language from early childhood.

</comment></comment><comment><USER4> There are also studies that suggest that people's morality changes depending on the language used. In particular, they become less utilitarian with a foreign language: [https://www.scientificamerican.com/article/how-morality- chan...](https://www.scientificamerican.com/article/how-morality-changes-in-a- foreign-language/)

</comment><comment><USER5> From the article title I thought this article would be why FORTRAN, LISP, and COBOL still survive after over 60 years of use. First, these computer languages started with a lot of good ideas and evolved to incorporate ideas from subsequent decades like structured programming and objects. Second, there are massive institutional code bases in these early languages that are too difficult to rewrite. These languages might even make it into the 22nd century.</comment>


Disinformation and 'fake news': Final Report published - <AUTHOR> https://www.parliament.uk/business/committees/committees-a-z/commons-select/digital-culture-media-and-sport-committee/news/fake-news-report-published-17-19/ <comment><USER2> The truth is the internet is teaching the biggest lesson ever in critical thinking and getting your information from many sources across spectrums, countries, divides and more to find out what is really going on.

People must think about why they are hearing about something and the layers and goals that are behind it and drive it.

Let's hope that people see these disinformation and misinformation efforts as a lesson and not somewhere they can bask in their confirmation bias all day, or make decisions based on fear, in those cases the populace is easy to manipulate, divide and conquer.

When something it too salacious or fits a narrative too perfectly, someone/group is marketing you in a direction and has you possibly in an active measure.

<comment><USER3> I think one of the lessons of the history "accelerate the contradictions" ( [http://acceleratethecontradictions.blogspot.com/2010/04/acce...](http://acceleratethecontradictions.blogspot.com/2010/04/accelerate- contradictions-notes-towards.html) ) is that putting the public in a situation where they have to improve lest there be a huge disaster, is a good way to get a huge disaster.

<comment><USER4> All the children cannot be above average.

<comment><USER5> They can if the children of tomorrow are compared with children of yesteryear, and children's education system has improved such that children of tomorrow excel at all the metrics measured.

For example, how much "smarter" are children today than children of 1000 b.c.?

</comment></comment></comment></comment><comment><USER6> I think censoring the "fake news" is far more dangerous than to just let it exist and have people think for themselves about what's true or false.

<comment><USER7> Why?

<comment><USER8> There is a theory that says people will be inoculated against fake news, if only they're exposed to enough of it. That critical thinking skills, like immune systems and website security, are made strong by unrelenting attacks.

After all, it's unlikely we'll ever see a day where everyone agrees on everything so completely that no two humans disagree about anything! And as long as that's the case, we'd better know how to deal with people telling us things that are wrong.

The theory goes that seeing social media messages with obvious lies like "Put nails in your tyres for better winter traction" and "Shoot yourself with small calibre bullets to build an immunity to larger bullets" teaches people scepticism and rational thought they can apply to much more complex statements like "inflation is caused by the fed printing money" and "the highest crime areas are those with a lot of immigrants"

Does it work? Have we seen it working hitherto? You be the judge!

<comment><USER9> Here in Brazil there is a very strong media monopoly, so most people are used to being fed only one side of the stories. It breaks critical thinking skills and leaves people completely vulnerable to trash content propagated within whatsapp and facebook.

The problem here is that these "propagandists" cater to people's emotions in a very persuasive manner. It's not that it will blatantly tell you "Shoot yourself with small calibre bullets", but rather it will inseminate the thought that perhaps you should shoot your neighbor, out of fear. The message is so masterfully targeted and gamified that even if 1 person starts having more "critical thinking" and simply shrug off the information, there are 10 more people ready and willing.

</comment></comment></comment></comment><comment><AUTHOR> PDF of the report [https://publications.parliament.uk/pa/cm201719/cmselect/cmcu...](https://publications.parliament.uk/pa/cm201719/cmselect/cmcumeds/1791/1791.pdf)

</comment><comment><USER10> Medical disinformation often rises to the top of search results:

[https://www.wired.com/story/the-complexity-of-simply- searchi...](https://www.wired.com/story/the-complexity-of-simply-searching- for-medical-advice/)

<comment><USER11> I found something like this when searched for antibiotic resistance in russian.

Mainstream, 70-years old boring truth is not the top result. Top result is fringe controversial study, that is reported all over recent news. This is so frustrating.

<comment> Const-me What did you expect? Antibiotic resistance is a new phenomenon. There's no science in modern Russia, and good sources on the subject like "The Lancet" and NIH's pubmed are in English.

<comment><USER12> I guess you'd expect that but it could in theory be fixed with better search algorithms. Perhaps AI that could distinguish The Lancet and crank stuff.

<comment> Const-me There's no Russian version of The Lancet, nor Russian translation. Nothing to distinguish crank stuff from.

People can distinguish if they speak other languages and search global Internet. Maybe I'm underestimating AI but I don't think AI can do that currently, nor will be able in foreseeable future.

</comment></comment></comment></comment></comment><comment><USER13>
       Calls for:
       Social media companies obliged to take down known sources of
       harmful content, including proven sources of disinformation

It will be interesting to see how Reddit addresses The_Donald or 8chan deals with the QAnon nonsense in the wake of this.

 _Edit: Feel free to disagree with the law, but you don 't do that with a downvote._

<comment><USER14> How do you prove something is disinformation? Time for a ministry of truth? I can imagine already imagine for example which version of the Convington kids case would be considered official and information.

<comment><USER15> This is exactly right. Trusting a government to determine what is true and what is not, and enforcing that version of truth via shutting down certain speech and penalising those who speak it, is a bad idea. I've no problem with a government announcing that they believe something is true or false, but I have a big problem with them restricting the speech of people who happen to disagree.

<comment><USER16> Instead we have a ministry of truth being perpetuated by sites such as Youtube, with incredibly harmful information being disseminated and treated as truth resulting in disease outbreaks and outright denial of history.

In your idealistic world of free speech, the innocent pay the price for the freedom to spread lies and deceit. At what point do we accept that it has to be fixed, or are we OK with diseases previously eradicated coming back in full force?

<comment><USER17> The difference is that there is also an overwhelming force of true speech to counter the lies and deceit. Once you start banning speech, only "official" lies will be allowed. See China.

Where the hell did you get the idea that free speech is only OK until people start to lie?

You can take my free speech from my cold, dead mouth.

<comment><USER16> If that overwhelming force of true speech worked, then why hasn't climate change denial, anti-vaccine nonsense etc been eradicated from the public sphere yet?

In fact, why are those movements growing?

<comment><USER18> That is because climate alarmism has been around for quite a while and anybody who has memory longer than a goldfish's can easily recollect the promised catastrophes which just didn't happen. Anti-vaccine nonsense is often painted as refusal to do all vaccines (there are nuts like that but they are few) rather than not exactly groundless suspicion that SOME new vaccines and aggressive vaccination regiments are pushed onto public not for the benefit of people but rather for the benefit of pharmas and because of evident willful misrepresentation of potential for dangerous side effects - not to mention that the very notion of forced medication strikes many people as utterly totalitarian.

These movements are growing precisely because of persistent lying, exaggerations, and totalitarian inclinations of people who promote climate "change" (a good example of Orwellian newspeak, isn't it?) and ever-growing number of must-do vaccines as reason for more taxation and more corporate welfare. A normal human reaction to being forced to submit is to resist. Even if force is supposed to be for the victim's own good.

<comment><USER16> > These movements are growing precisely because of persistent lying, > exaggerations, and totalitarian inclinations of people who promote climate > "change" (a good example of Orwellian newspeak, isn't it?)

What complete and utter bullshit. Yes, not all of the predictions behind climate change have come true; science evolves and new predictions or estimations are made as advancements in technology grow. Yet we've seen many of the predictions come true and many far worse things start to pop up, including the mass decline of insects worldwide and the rapidly rising sea level.

<comment><USER19> _" we've seen many of the predictions come true... including the mass decline of insects worldwide"_

Interesting you should mention that. No doubt you're referring to the recent study [1], heavily pushed by the likes of The Guardian [2], the Washington Post and the BBC, warning of the "collapse of nature" due to declining insect population.

Did you also know that this study turned out to be flawed to the point of uselessness? So flawed in fact that the Global Warming Policy Foundation sent a complaint to its publishers and is calling for its withdrawal? [3]

I think you unwittingly just proved aveross' point vis-à-vis 'climate alarmism'...

[1] [https://www.pnas.org/content/115/44/E10397#F1](https://www.pnas.org/content/115/44/E10397#F1)

[2] [https://www.theguardian.com/environment/2019/feb/10/plummeti...](https://www.theguardian.com/environment/2019/feb/10/plummeting- insect-numbers-threaten-collapse-of-nature)

[3] [https://www.thegwpf.org/hyper-alarming-study-was- hype/](https://www.thegwpf.org/hyper-alarming-study-was-hype/)

<comment><USER20> Hang on just a bloody minute.

The recent study "heavily pushed by the likes of the Guardian" (sic) refers to _an entirely different_ meta study[1] of 73 global reports. No mention of or link to the older study you cite.

The meta study concludes intensive agriculture is the main driver of the declines, particularly the heavy use of pesticides. Urbanisation and climate change are also significant factors. So the failure or otherwise of the temperature record in a single location is mostly irrelevant to their findings. If it is even included in the analysis at all.

What a delightful illustration for a thread on disinformation and fake news.

[1] [https://www.sciencedirect.com/science/article/pii/S000632071...](https://www.sciencedirect.com/science/article/pii/S0006320718313636)

<comment><USER19> _" No mention of or link to the older study you cite."_

It's right there in the 4th paragraph.

<comment><USER20> Which is link to a previous article, so adds a level of indirection. If you'd cited that in the first place, fine.

Even if the complaint letter turns out to be upheld (unlikely given the history of the authors, but you never know), it doesn't affect, or even question, the conclusion of the meta study in the Guardian piece you _did_ link to. The one that concludes climate is only a significant secondary factor.

Until said letter is upheld as finding a valid flaw in the study, it's worthless. That's how science progresses. Individual studies are found accurate or wanting, further research confirms, disputes or adds precision.

A few steps from the hyperbole of "flawed to the point of uselessness", as though it is already proven.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER21> The problem is and always will be that truth is tricky to define, and "blocking fake news" is immensely easy to exploit for nefarious purposes.

Perhaps the better solution here is not to try to block the flood but to manipulate it by using market incentives. Notice how high-quality journalism tends (again, quite subjective) towards subscriptions and sponsorships, as users will actively want to participate whereas lower-effort content has to lean heavily on black-box advertising networks. This could be solved with a few changes: tax advertising a lot more, and use these funds to subsidise subscriptions and sponsorships in media.

</comment><comment><USER22> Snippets from 2014 CFR discussion:

"…So the question I'm asking myself is can we figure out a way to accelerate the second curve of human response in a fashion -- in -- in -- in identical ways to the way DARPA accelerates the first curve of technological challenges."

"… What about Facebook? What about Twitter? I have no idea what Twitter is good for. But if it flips out every tyrant in the Middle East, I'm interested."

Funny how tools work…

<comment><USER23> Source on this?

<comment><USER22> [http://web.archive.org/web/20140531073143/http://www.cfr.org...](http://web.archive.org/web/20140531073143/http://www.cfr.org:80/technology- and-foreign-policy/technology-policymaking-future/p33025)

</comment></comment></comment><comment><USER24> is it me or does the committee seem lazy? their report is basically a collage of NYtimes articles, which they quote as evidence , even when it is speculative

their facebook emails 'evidence' isn t particularly related to their conclusions either. i mean facebook has been treating its small publishers unfairly since the age of zynga, just search facebook's forum to find developers actively calling them out. it's not really related to disinformation.

i would expect a little more work based on their own secret services about the extent of russian trolling in their own referendum.most of all i would expect at least some evidence that these campaigns can change people's opinion. not liking an electoral outcome cannot be used as evidence that they were effective

that said, i do hope they ban political advertising to death, because the ones who benefit from it are the politicians who write this report themselves. however, regulating the truth? i thought george orwell was british

<comment><USER3> > basically a collage of NYtimes articles

This is such an inaccurate statement that I'm tempted to joke about it being fake news.

This report is about as serious as it gets in terms of professional investigations. There's almost 100 pieces of written evidence and they called a number of witnesses to give primary evidence (famously including Zuckerberg who refused to show up). Where they have referred to the NYT, it is properly cited in the footnotes. Eight of the 349 footnotes refer to the NYT. Care to retract your accusation of laziness?

> ban political advertising to death

It largely is, in the UK. Certainly by American standards of "free speech". There's one huge exception: Facebook adverts were unregulated...

<comment><USER24> i was actually expecting something more related to the UK and brexit. Particularly, evidence that misinformation can change or has changed the minds of people regarding the EU, or brexit in general . Instead this seems to be a hodgepodge of well know information about facebook and cambridge analytiuca and the US elections. they use some very speculative quotes from nytimes such as this

> allegedly to spread anti-semitic information about George Soros and his > campaigning activities, afer Mr Soros called Facebook "a menace to society" > in early 2018.

and some very spurious data such as this

> is interesting to note that, as of 30 November 2018, the online Government > response to our Report received a total of 1,290 unique page views and the > PDF has been visited 396 unique times from the website.265 In the month > following its publication, over 63% of views of the report online were from > foreign IP addresses (whereas, on average, 80% of viewers of Reports are UK- > based), and of these, over half were from Russia

in which they seem to be downgrading russian interference to spam bots. Their own conclusion with regards to interference in UK elections is that "Te Government should be conducting analysis to understand the extent of Russian targeting of voters during elections". i.e. "we don't know"

that said i m not disputing the accuracy of what is stated. it's just not enough, and not surprising at all

</comment></comment></comment><comment><USER25> > Compulsory Code of Ethics for tech companies overseen by independent > regulator

I would have loved to see strong moves in favor of critical thinking, rational argumentation, scientific analysis/study.

The suggested actions will leave us with a government that dictates what it believes is good ethics and penalizes everyone who does not agree - Not quite the opposite of what I'd hope for, but its scarily close.

<comment><USER26> As a European I would be strongly in favor of the UK leaving the union as fast as possible if anything of that kind gets implemented.

<comment><USER3> The UK is currently subject to less of this kind of regulation than several other European countries.

<comment><USER26> The UK has been on a severe surveillance and control trip for over a decade now.

</comment></comment></comment></comment><comment><USER27> The problem is not fake news. The problem is extremely biased news.

Is everything Trump does is great? Hell, no.

Is everything Trump does is stupid? Probably not either.

But will hardly ever hear from CNN that "actually, Trump's idea here was good, let's give him a credit".

Same goes the other way round.

</comment><comment><USER28> The truth is that there has always been misinformation, lies, and manipulation through media in politics.

Just look at the history of TV and radio regulation in the US and Europe. Look at how so many future leaders moved from media careers into politics.

The problem is that this time it didn't work as usual and the process had the wrong outcome (Brexit and mr. Trump). So now "we" are looking for excuses and reasons to expand government power.

</comment><comment><USER29> Do we need a trust system of some kind? Maybe journalists should be held accountable in the same way that doctors, structural engineers, architects etc. An article should be cryptographically signed by the journalist. If a journalist publishes incorrect information, report it to their professional body. Factual inaccuracies should be corrected in the reported article or the journalist should withdraw their certificate from it. Several egregious uncorrected factual innacuracies, as judged by a jury of their peers and lay people, causes a journalist to be struck off and their certificate revoked forever lableing everything they have written as questionable. Protect the word 'News' in law so that you can't use it without having certified journalists writing the copy. People can still publish and read anything they want just like you are free to go to a homeopath if you want, but there would be a trust structure in place.

<comment><USER24> a blockchain trust system for reporters (i think the profession of journalist has come full circle and it's time to retire the label)

</comment></comment><comment><USER16> Considering the amount of damage anti-vaccine hysteria and disinformation has caused, I can't say I disagree with their conclusions. If debating them with facts and logic actually worked, then we should've seen the movement fade away years ago.

It's not, and it's growing as a result of faulty search algorithms being used by people to spread conspiracy theories and fake nonsense. It's exactly why flat earth nonsense is spreading as well [1].

[1] [https://www.theguardian.com/science/2019/feb/17/study- blames...](https://www.theguardian.com/science/2019/feb/17/study-blames- youtube-for-rise-in-number-of-flat-earthers)

<comment><USER30> The media has its own disinformation. For example the wage gap « at equal work » is the biggest statistical error ever made (never were basic criteria taken into account), thousands of researchers voiced that the way the media presented it made people think women were paid less for the same work...

...and yet researchers couldn't get their voice to be heard. Given the number of unfair laws and bylaws that were passed on the understanding that women were paid less, the wage gap has been a lie at a huge scale.

<comment><USER31> I regularly quote said statistic - can you link me some reliable sources supporting your claim?

<comment><USER32> The key word in such research is called "Adjusted pay gap" vs "Unadjusted pay gap".

Differences in hours worked, occupations chosen, education, job experience, and location are the big ones. Then there are a few additional ones like age and health.

Depending on which study you either end up with a very minor (~5%) difference between women and men, or none.

With unadjusted pay gap that chooses to limit the selection to fully employed you get a number around 60% in the US. If you don't limit and count all citizen you get a weird number, and if you look at the problem from a social economic status perspective you get an even weirder results.

<comment><USER31> That is really interesting, and not what I had been lead to believe, but could you provide me with any reliable sources to this affect?

</comment></comment></comment></comment></comment><comment><USER33> Fake news is new media. It must be really important if google news has a special section dedicated to it on the right column where you can see it without needing to scroll down the page. The uk has the bbc to spread its agenda, mostly in favour of further colonisation and to protect its weapons supply network across the middle east , and doesnt want the competition. Fake news is actually competition that makes the 'real' news try harder.

<comment><USER34> > further colonisation

What? UK has been de-colonising for decades. Other former colonial powers held on much harder, e.g. France.</comment></comment>


YC Summer 16 Applications open - <AUTHOR> https://apply.ycombinator.com/ <comment><AUTHOR> The only reason I posted is because it was part of the Fellowship announcement, which some people may not have read.</comment>