Airflow and the Future of Data Engineering: A Q&A - <AUTHOR> https://medium.com/the-astronomer-journey/airflow-and-the-future-of-data-engineering-a-q-a-266f68d956a9 <comment><USER2> _Airflow. This framework is used by numerous companies and several of the biggest unicorns — Spotify, Lyft, Airbnb, Stripe, and others to power data engineering at massive scale._

Is that correct? I've been using (and enjoying) Luigi[1] which came out of Spotify. I haven't seen anything about them switching to Airflow.

Edit: Now I see in the interview there is this:

 _About Luigi, it is simpler in scope than Airflow, and perhaps we're more complementary than competition. From what I gather, the main maintainer of the product has left Spotify and apparently they are now using Airflow internally for [at least] some of their use cases. I do not have the full story here and would like to hear more about it. I'm thinking that many of the companies choosing Luigi today might also choose Airflow later as they develop the need for the extra set of features that Airflow offers._

But there are 2 day old commits in the Luigi directory, so I don't know. I like Airflow too, but it did seems a lot more complicated the Luigi when I played with it.

[1] [https://github.com/spotify/luigi](https://github.com/spotify/luigi)

<comment><USER3> If simplicity and non-Python-centricity matter, I encourage folks to look into Digdag [1][2].

It's Ansible for Workflow Management.

While both Luigi and Airflow (somewhat rightfully) assume the user to know/have affinity for Python, Digdag focuses on ease of use and helping enterprises move data around many systems.

If we learned one thing from today's S3 outage, it's not enough to use multiple cloud infrastructure providers: you should probably have your data in multiple cloud providers as well.

[1] [https://www.digdag.io](https://www.digdag.io)

[2] [https://github.com/treasure-data/digdag](https://github.com/treasure- data/digdag)

<comment><USER4> [auhtor] Oh cool, I didn't know you guys released your solution yet, I remember demoing Airflow to you guys early on. Looks like it turned out great, congrats on the release!

<comment><USER3> Thanks! Airflow was/has been a great source of inspiration =)

</comment></comment></comment></comment><comment><USER5> One of the important paradigms that I think Luigi and Airflow miss is that they treat pipelines as a DAG of tasks, when it really should be thought of as a DAG of Data.

It's a subtle difference, but has huge impacts when you're trying to dynamically scale tasks based on cluster resources and track data lineage throughout your system.(Disclosure: I'm the founder of Pachyderm[0], a containerized data pipeline framework where we version control data in this way).

Check out Samuel Lampa's post[1] about dynamically scaling data pipelines for more details.

[0] github.com/pachyderm/pachyderm [1] [http://bionics.it/posts/dynamic- workflow-scheduling](http://bionics.it/posts/dynamic-workflow-scheduling)

<comment><USER4> [Airflow author] The task is centric to the workflow engine. It's an important entity, and it's complementary to the data lineage graph (not necessarily a DAG btw).

At Airbnb we have another important tool (not open source at the moment) that is a UI and search engine to understand all of the "data objects" and how they relate. It includes datasets, tables, charts, dashboards and tasks. The edges are usage, attribution, sources, ... This tool shows [amongst other things] data lineage and is complementary to Airflow.

<comment><USER5> Does the other tool you're talking about that works with Airflow allow you to scale your Airflow tasks based on, for example, the amount of new input data that needs to be processed? That was one of the major challenges we see in bioinformatics workloads. Sometimes you have a few new samples to run and other times there are thousands -- so your task scheduler, although it is centric, needs to have an understanding of the data too.

<comment><USER4> At a high level [for Airflow specifically] the scheduler or workflow engine cares most about the tasks and their dependencies, and is somewhat agnostic about the units of work it triggers.

It's possible to use a feature called XCom as a message bus between tasks, but would typically direct people in the direction of having stateless, idempotent, "contained" units of work and avoid cross task communication as much as possible. [https://airflow.incubator.apache.org/concepts.html#xcoms](https://airflow.incubator.apache.org/concepts.html#xcoms)

For your case [which I have little input on] I think singleton DAGs described in another post on this page may work.

</comment></comment></comment></comment><comment><USER6> Hey HN, I co-wrote the article with Maxime. A little late to the party here but happy to answer any questions you might have about it. I'll send him the link as well.

 _By the way, thank you to Maxime for sharing his thoughts, and the Astronomer team for contributing great questions._

<comment><USER4> Maxime reporting for duty here, I'll go through the thread and answer questions / add comments.

</comment></comment><comment><USER7> Airflow works well for "static" jobs, but I miss something like airflow for dynamic jobs.

By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.

<comment><USER8> I've started working on a code generator for Airflow. Not primarily because I needed dynamic jobs, but more because I didn't want to keep writing the Airflow boilerplate.

I imagine I'll eventually need to add some sort of management system to move these dynamic jobs in and out of Airflow to keep them from bloating the database or cluttering the UI.

<comment><USER6> I'd be interested in hearing more about your project and if you plan to open source. Are you using something like Cookiecutter or Yeoman, or a different level of abstraction?

<comment><USER8> My effort is a collection of custom operators for operations that we use very commonly (run a query, python script, or mapreduce job) and a code generator that takes a simple text spec describing a set of operations and generates the Python DAG script. I'm not familiar with Cookiecutter or Yeoman but generating Python code myself hasn't been complicated. Open-sourcing it isn't an option right now with my current employer.

<comment><USER6> Thanks, this sounds really interesting. If this changes or you decide to write personal code with similar structure, I'd definitely like to discuss further. I'm also currently exploring DAG structure but with class and function abstractions with the goals of increasing code reuse and minimizing boilerplate around our custom operators.

Cookiecutter takes a Python code file as input with Jinja interspersed (the template). When you want to make a new instance from the template, it gives command-line prompts, and then evaluates the Jinja logic — custom variables, loops, etc to output Python code. It took me 15–30 minutes to get started and has already paid off.

</comment></comment></comment></comment></comment><comment><USER9> We're currently in a PoC phase of implementing Airflow and testing it out versus Luigi. So far, what I've liked, is that Airflow seems to be much more extensible and modular than Luigi. Getting Luigi to play nicely with our particular set of constraints was painful, and subclassing the Task was also tricky because it was way more opinionated about the structure of the class. Airflow seems way less so. There also seems to be way more right out of the gate in terms of built in task types. And the UI looks nicer.

We haven't made our final determination yet, but Airflow at the current moment feels better.

<comment><USER4> [Airflow author here] one of the main differences between Airflow and Luigi is the fact that in Airflow you __instantiate __operators to create tasks, where with Luigi you __derive __classes to create tasks. This means it 's more natural to create tasks dynamically in Airflow. This becomes really important if want to build workflows dynamically from code [which you should sometimes!].

A very simple example of that would be an Airflow script that reads a yaml config file with a list of table names, and creates a little workflow for each table, that may do things like loading the table into a target database, perhaps apply rules from the config file around sampling, data retention, anonymisation, ... Now you have this abstraction where you can add entries to the config file to create new chunks of workflows without doing much work. It turns out there are tons of use cases for this type of approach. At Airbnb the most complex use case for this is around experimentation and A/B testing.

I gave a talk at a Python meetup in SF recently talking about "Advanced data engineering patterns using Apache Airflow", which was all about dynamic pipeline generation. The ability to do that is really a game changer in data engineering and part of the motivation behind writing Airflow the way it is. I'm planning on giving this talk again, but maybe I should just record it and put it on Youtube. It's probably a better outlet than any conference/meetup...

</comment></comment><comment><USER10> How relevant are Airflow and similar to those of us who aren't operating at unicorn scale but are shuffling hundreds of CSVs & Excels and wrangling RDBMS with SQL?

<comment><USER11> I have used Talend in the past, sounds like a fit. But this seems to fit a different need around job management.

<comment><USER10> Talend makes my teeth grind. I don't understand why an ETL tool uses a strongly typed language for a foundation. The number of fun productive hours I've spent swapping chars to varchars, int to decimal & vice versa. In 2017 computers can read a registration plate from a blurry photograph and spot a criminal in a stadium, but a user puts apostrophe in a CSV file and schmoo leaks everywhere.

<comment><USER12> Strong typing helps. Keep in mind enterprise ETL tools are designed to move data from oltp to olap databases, which are often strongly typed as well.

<comment><USER10> When a tool insist on you cast char fields to varchars before you can test two fields for equality, or keeps changing all your decimals to floats, how is that helping? I'm saying that if the underlying language was loosely typed, those kind of productitvy saps & bug fountains would not happen. In the few instance where you cared about type, a loosely typed language can usually offer something.

Last time I checked, enterprise ETL tools were sold as capable of a lot more than simple OLAP to OLTP. I find the reality provided is somewhat underwhelming. Given that facebook can tell the difference between a photo of Dave and one of Jim, why do I have to manually provide a mask for every single date field flowing through an enterprise?

<comment><USER13> I don't use enterprise ETL tools. I pretty much write my own every time and occasionally I'll supplement that with things like Kiba.

</comment></comment></comment></comment></comment></comment><comment><USER14> [Bias Alert: I'm Head Chef of DataKitchen]. Our perspective is that the DAG abstraction should not apply only to data engineering, but the whole analytic process of data engineering, data science, and data visualization. Analytic teams love to work with their favorite tools -- Python, SQL, ETL, Jupyter, R, Tableau, Alteryx, etc. The question is how do you get those diverse teams and tools to work together to deliver fast, with high quality, and reusable components?

We've identified seven steps taken from DevOps, CI, Agile and Lean Manufacturing ([https://www.datakitchen.io/platform.html#sevensteps](https://www.datakitchen.io/platform.html#sevensteps)) that you can start to apply today. We also created a 'DataOps' platform that incorporates those principles into a software: [https://www.datakitchen.io](https://www.datakitchen.io).

The challenge is that there are many separate DAGs (and code and configuration) involved in producing complete production analytics embedded in each of the tools the team has selected. So what is needed is a "DAG of DAGs" that encompasses the whole analytic tool chain.

<comment><USER4> [Bias Alert: author of Airflow] can confirm that Airflow allows you to incorporate all of the seven steps, and more as an open platform.

At Airbnb Airflow is far from being limited to data engineering. All the batch scheduling goes through Airflow and many team (data science, analysts, data infra, ML infra, engineering as a whole, ...) uses Airflow in all sorts of ways.

Airflow has a solid story in terms of reusable components, from extendable abstractions (operator, hooks, executors, macros, ...) all the way to computation frameworks.

</comment></comment><comment><USER15> [Disclaimer: I work for Composable] My team and I are working on a project that I would consider a competitor to Airflow. I'm not overly familiar with Airflow, but Composable seems to be fit for a much wider variety of use cases.

In Composable's DAG execution engine, you can pull in data from various sources (SQL, NoSQL, csv, json, restful endpoints, etc.) into our common data format. You can then easily transform, orchestrate, or analyze your data using our built-in Modules (blocks) or you can easily write your own. You can then view your resulting data all within the webapp.

Reading the comments, it seems like Composable supports a lot of the things people are asking for here that Airflow is lacking. Maybe check us out and let us know what you think!

For more information: Composable Site - [https://composableanalytics.com/](https://composableanalytics.com/) Try it yourself - [https://cloud.composableanalytics.com/](https://cloud.composableanalytics.com/) Composable's Blog - [http://blog.composable.ai/](http://blog.composable.ai/)

<comment><USER4> [author of Airflow here] as I wrote in another comment, I'd argue for a programmatic approach to workflows/dataflows as opposed to drag and drop. It turns out that code is a better abstraction for software: [https://medium.freecodecamp.com/the-rise-of-the-data- enginee...](https://medium.freecodecamp.com/the-rise-of-the-data- engineer-91be18f1e603)

I'd also argue for open source over proprietary, mostly to allow for a framework that is "hackable" and extensible by nature. You can also count on the community to build a lot of the operators & hooks you'll need (Airflow terms).

<comment><USER16> You can use Composable's Fluent API to author and run dataflows in code. No GUI required. Composable's platform is open and completely extensible in sense that anyone can add applications or first class modules to the system, including swapping out some of the more internal components.

</comment></comment></comment><comment><USER17> A lot of name dropping and unprovable statements. I'm really interested by the domain and progress, but Airflow needs to be more generous in _real_ information and less in marketing bs. Can someone share a more introductory article about what makes Airflow different from the current state of the art?

<comment><USER18> Here is, a slightly outdated, article that compares several ETL workflow tools [http://bytepawn.com/luigi-airflow-pinball.html](http://bytepawn.com/luigi- airflow-pinball.html) . Why we choose Airflow was because of the following reasons:

* Scheduler that knows how to handle retries, skipped tasks, failing tasks

* Great UI

* Horizontal scaleable

* Great community

* Extensible; we could make it work in an enterprise context (kerberos, ldap etc)

* No XML

* Testable and debug-able workflows

</comment></comment><comment><USER19> +1 for Airflow. I use it every day to handle tasks with many components and dependencies. I love that everything is code & version-controlled.

I do wish it had a REST API though.

<comment><USER20> I've been keeping close tabs on the project for a while now and it seems that version 1.8, which should be released in a few days, has the beginning of a rudimentary API. It also looks like more endpoints are being planned for subsequent releases

</comment></comment><comment><USER21> Data engineering is converging under the umbrella of DataOps. For those interested, there's a DataOps Summit in Boston this June [https://www.dataopssummit.com/](https://www.dataopssummit.com/)

<comment><USER22> This is just Data Management, a term which predates "DataOps" by more than a decade in both research and enterprise. I don't really think it needs a rebranding.

<comment><USER21> That's not really the case... There's a nice, short podcast that was just posted that defines DataOps more broadly - spend 20 minutes listening to it, and see what you think. Interested to hear your thoughts.

[https://thenewstack.io/delving-dataops- matters/](https://thenewstack.io/delving-dataops-matters/)

</comment></comment></comment><comment><USER23> I'm used to running R scripts with cron to handle some batch jobs and I'm no python dev. Would it be easy to start using Airflow?

<comment><USER18> DAGS in Airflow can just be a few lines. Some understanding of the syntax of python is required. But you can start simple and add complexity as you require it.

<comment><USER9> To add to that, you can create a DAG class that's really just a wrapper that executes your R script and waits for its return code.

</comment></comment></comment><comment><USER24> Luigi is nice, because it is really simple to get started and it gradually allows you to do more complex things like (custom) parameter types, custom targets, enhanced super classes and dynamic dependencies, event hooks, task history and more.

One thing that I missed a bit was automatic task output naming based on the parameters of a task, so I wrote a thin wrapper for that [1]. This helps, but mostly for smaller deployments.

Airflow and luigi seemed to me like two side of the same thing: fixed graphs vs data flow. One fixates the DAG, the other puts more emphasis on composition.

That said, I am excited about the data processing tools to come - I believe this is an exciting space and choosing or writing the _right tool_ can make a real difference between a messy data landscape and an agile part of business and business development.

[1] [https://github.com/miku/gluish](https://github.com/miku/gluish)

<comment><USER5> Definitely agree that that is one of the great points with Luigi. Airflow's UI of course blows everything out of the water IMO.

As for organizing your data, my personal and very biased opinion is that version control semantics similar to Git [0] are a pretty good way to help tame the complexity of ever-changing data sets. We already version code, but with versioned data too, now everything becomes completely reproducible.

[0] Our Data Science Bill of Rights: [http://www.pachyderm.io/dsbor.html](http://www.pachyderm.io/dsbor.html)

<comment><USER24> The git angle would be a huge step forward. What I found is that reproducability is not always on people's mind when they designing such systems, whereas I believe it's one of the most important properties.

Pachyderm is on my TODO list for a while, so thanks for reminding me, I'll try to implement something real with it soon.

</comment></comment></comment><comment><USER25> Minor gripe - why can I not execute an entire DAG (end to end) from the UI? Also trying to execute single tasks from the UI using the "run" functionality gives a CeleryExecutor requirement error... sorry, I know this isn't the help forums but it sounds like the most trivial tasks were overlooked.

<comment><USER26> Actually you can but it is a bit clunky. Go to Browse > Dag Runs and select the 'Create' tab. This pulls up a form where you type in the Dag ID, enter the start time (set to now but keep in mind it is the local time of the web server), and a Run ID.

Totally agree there should be both a simple Start Run and Stop Run button.

<comment><USER6> What I've been doing when debugging is flipping the DAG switch to off, clearing the first task in the DAG so the whole thing re-runs, then back to on.

</comment></comment></comment><comment><USER27> Disney Studios uses Azkaban because it's language agnostic, and it believes that in the data space there is a huge advantage to static (and strong, but that's beside the point in this argument) typing in the data space.

The language agnostic aspect means that non software engineers can also use the orchestration platform for runbook automation.

</comment><comment><USER28> MHO There seems to be quite some conceptual overlap between Airflow's DAG and Spark RDD.

It seems to me that Airflow is Spark-on-a-db ... or rather Spark is Airflow- on-Hadoop.

Does anyone know what the difference is ?

<comment><USER29> Airflow doesn't have anything to do with data storage, movement or processing. It's a way to chain commands together in such a way so that you can define "do Z after Y a Z finish", for example. Many people use it like a nice version of cron with a UI, alerting, and retries.

<comment><USER28> so - celery + spark ? or just Celery Canvas ? ([http://docs.celeryproject.org/en/latest/userguide/canvas.htm...](http://docs.celeryproject.org/en/latest/userguide/canvas.html))

P.S. I'm not trolling - I'm genuinely trying to get a sense of why and when would I use Airflow. Is it a point of scalability, of productivity , etc ?

For example - the positioning of spark is simple: scalability. Celery is also very clear: simplicity with good enough robustness if using the rabbitmq backend.

what does Airflow do differently ?

<comment><USER4> In a modern data team, Spark is just one of the type of job you may want to orchestrate. Typically as your company gets more tangled in data processing, you'll have many storage and compute engines that you'll have to orchestrate. Hive, MySQL, Presto, HBASE, map/reduce, Cascading/Scalding, scripts, external integrations, R, Druid, Redshift, miroservices, ...

Airflow allows you to orchestrate all of this and keep most of code and high level operation in one place.

Of course Spark has its own internal DAG and can somewhat act as Airflow and trigger some of these other things, but typically that breaks down as you have a growing array of Spark jobs and want to keep a holistic view.

<comment><USER28> that is an incredibly lucid answer. That should be the first line on the Airflow project.

</comment></comment></comment></comment></comment><comment><USER30> Amazing framework with a lot of functionalities. A tool built to be extensible just what an open source software should look like :)

</comment><comment><USER31> How would you see Airflow in relation to Apache Beam / GC Dataflow?

<comment><USER4> [author] Airflow is not a data flow engine, though you can use it to do some of that, but we typically defer on doing data transformations using/coordinating external engines (Spark, Hive, Cascading, Sqoop, PIG, ...).

We operate at a higher level: orchestration. If we were to start using Apache Beam at Airbnb (and we very well may soon!), we' use Airflow to schedule and trigger batch beam jobs alongside the rest of our other jobs.

<comment><USER31> Thanks, that's really interesting. The usage of 'pipeline' to describe both sorts of system made me think there was a lot of overlap, but I'm understanding now how they are complementary.

</comment></comment></comment><comment><USER32> Who has used both Airflow and Spinnaker? Feedbacks?

</comment><comment><USER33> The actual article is here: [https://medium.com/the-astronomer- journey/airflow-and-the-fu...](https://medium.com/the-astronomer- journey/airflow-and-the-future-of-data-engineering-a- q-a-266f68d956a9#.xtu264vu2)</comment>


Ask HN: How do I get my life back in order? - <AUTHOR> I'm a 25 year old for whom the internet has been a long time source of fascination and joy. My parents weren't well off and couldn't afford to send me through university so I decided to study on my own. For a self taught engineer who lived in a place where nothing extraordinary ever happened, I lucked out and worked for big name brands, with amazing people and made good money. But a year and a half ago, the most unfortunate thing happened and I lost my childhood friend to cancer. We did everything together, from racing toy cars as a child to hanging out at our favorite Persian restaurant as grownups, we were best of friends for 23 years. And needless to say his death destroyed me.
The first few months I cried every other day, I would scream my lungs out and cry in my car. It helped to cry, after which I could get on with the rest of my day. The crying eventually gave way to a need to rebuild my life. I took every opportunity to be social, exercise and do good work, all of which made me happier and my life was starting to look good again.
But in recent months, I have been losing motivation and the activities that used to give me joy no longer do anything for me.
Nothing interests me anymore. I am no longer motivated to solve problems, make money or do anything really. I just work enough get a paycheck, binge watch TV and occasionally go out with friends. I have become unproductive, feel constant anxiety and regret.
My judgement has become cloudy and I have trouble making even everyday life decisions. I find myself making decisions that I in hindsight usually regret. Even with all the agonizing, overthinking, reasoning, coin tossing and deliberating, I find myself making wrong decisions. The regret and failure of that decision manifests as an anxiety attack. I am tired of feeling this way!
How do I get over decisions that I know I screwed up on? How can I think more clearly and make better decisions?
How do I get my life back in order? <comment><USER2> Here is my perspective on some of what you have asked. I heard a great quote the other day, "don't look back, you aren't going there". Generally human beings do the best they can with the info they have available at the time. Respect and accept that. Believe me, I have been in your shoes having lost my partner, soulmate and beloved wife at 37 to rectal cancer. I can only say it gets easier to carry the burden. Although there will be moments when the survivor guilt is overwhelming, try to honor their legacy and the years they were robbed of by living as fully as possible. It will benefit you and the world around you. A good therapist will help too epecially one who has a specialty in cancer related. Be well. --A fellow traveler on the grief journey.

</comment><comment><USER3> Psychiatrist for some temporary help (unless this turns out to be an issue that has long been masked) and a therapist immediately. Don't wait assuming it'll just get better. The moment you finish making your appointment, grab some Nike's and start running. There's known formulas to fix the misery. I speak from a lifetime of struggling with depression. Don't ignore it. Don't "get used to it" like I've heard others say. Fix it.

<comment><USER4> This, definitely. There's no shame in seeking help when you need it. It can be very hard to lift yourself back up. I strongly encourage you to seek a psychiatrist's help ASAP.

</comment></comment><comment><USER5> My condolences on your loss.

As others have suggested, talking with a qualified clinical therapist may be a reasonable way to coming to find a peaceful place for your friend's death and to live with your grief in healthier ways.

A qualified clinical therapist can bring understanding and insight to the grieving process. This doesn't mean that your loss isn't unique, just that people tend to encounter similar patterns when grieving and these patterns are normal and can be dealt with in healthy ways without ignoring the reality of personal loss.

My childhood best friend, Leonard, was struck and killed by a car Thanksgiving night, 1984. Writing that made me sad. The loss is still there after all those years. But I've found a place for it and it would be worse I think if I didn't still feel the loss. My friend Phil was in the Marine Corps when Leonard died. He'd known him longer than me. The Corps would not grant him leave for the funeral.

All these years later, sometimes when we talk, we talk about Leonard's death because it hasn't gone away. But mostly we talk about our families and friends and our work and the joy in our current lives. Because that doesn't go away either.

Good luck.

<comment><AUTHOR> I'm very sorry for your loss. It's hard moving on without your friend by your side.

<comment><USER5> Thanks. It was hard for me for a while.

I wouldn't say that the Leonard shaped hole in my life has become smaller. It's more that my life expanded to places beyond the old neighborhood, into interesting work, alongside a spouse and then a child, and always some dogs, and the hole is a smaller part of my whole life on the day to day, without changing the degree to which the hole is part of who I am.

</comment></comment></comment><comment><USER6> It sounds very, very much like you are depressed. You should seek medical advice.

I don't think there's much else to say, really. Get well, and the rest will follow.

</comment><comment><USER7> Tony Robbins' 'Strategy, State, and Story' framework works for me. Let me briefly explain this using my own terminology. Tony goes into this in much more detail, and I would recommend watching one of his more recent Youtube talks.

Strategy: There are very basic strategies for getting tasks done these days now more than ever, and they're all well documented, with people even outlining the potential pitfalls and benefits of the strategy. If your goal is to make friends, there is a strategy for that: Take up salsa classes, or a cookery course. If it's to make a million dollars, there are blueprints for that: Set up vending machines in every store you can find selling bubblegum.

State: State actually comes first before strategy and the idea is to get your mindset right before doing any task. Remember it's body first, and then the mind. The mind essentially runs off the body, so you want to nourish it. Eat healthy fats, avocados. Do Yoga. Make your gut biome more diverse with probiotics. Anything to make tasking seem more frictionless and effortless.

Story: Story is tied in with identity, and so much of our identity can be hacked. Your story doesn't have to be set in stone. Sometimes its useful to throw out your current narrative and replace it with an entire new one. Also: Nobody actually cares about your story. If you're telling yourself the same things over again and again in your head, you will start to believe it, so tell yourself what you want to hear.

</comment><comment><USER8> First of all, try not to beat yourself up about screwing up or making the wrong decisions. This is all normal and a part of your experience. What you have gone through may need some time to process.

Start with small steps. Over thinking and over reasoning is a way for your brain to avoid what you are actually feeling. This is normal because we are all inclined to avoid unpleasant feelings. Slowly try to get in touch with those feelings, even if they're painful. The more you try to avoid them, the more they will linger, and the more you try to suppress them, the deeper your depression. Getting in touch with those painful feelings are actually not as bad as you fear. It can actually be cathartic.

Along with all the suggestions to see a qualified therapist, I would recommend reading some of Pema Chodron's books. There's one in particular that may be relevant for your situation called 'When things fall apart'

I won't wish you good luck, because luck will not make things better for you. This will take some time and a little effort to work through. You have already taken the first step by reaching out. What you do next, and the skills and lessons you will learn through processing this experience will grow you as a person. Remember - one step at a time.

</comment><comment><USER9> I know religion is not something one is supposed to write about on HN, but this is a personal help issue and not a technical/social discussion, so I hope I will be forgiven.

But seriously: my advice to you is to find a pastor/cleric you can trust and talk to him about your situation (many are in fact trained grief counselors). I too lost a good friend some years back, and my faith and my relationship to God were really what brought me through the time of mourning. Please at least consider it.

</comment><comment><USER10> Get your ass to therapy--it's good for everyone.

Consider going to school and studying something perhaps unrelated to engineering. You'll meet people and be given a perspective you won't pick up on the internet.

Go outside, start working out (walking, jogging, lifting, swimming, rowing). Do something in meatspace.

</comment><comment><USER11> First, my condolences on your loss. Losing someone so close can be devastating. I have lost someone very close. Their death sent me into a downward spiral where I questioned everything around me. My family, job, friends, like, why am I here, etc.

I saw a psychiatrist. I took medication. We talked. It helped. I stopped the medication 10 years ago. I am thankful I admitted I needed someone to help me. I wasn't qualified to help myself and I didn't see the "light at the end of the tunnel" without them.

You can't right any wrongs until you understand why you wrong'd in the first place. People will understand.

Good luck. If you need a helping hand getting started this community can help.

</comment><comment><USER12> Sorry for your loss. It sounds like you are depressed and should chat with a therapist. Good luck.

</comment><comment><USER13> I agree that it seems like depression. It is really common after any major life event - good or bad stress. In this case, death. I'm sorry for your loss.

I'd recommend therapy and a trip to the doctor. If it makes it easier, call your physician and take this with you to explain. He or she might be able to recommend a psychiatrist or therapist for you as well.

If medication is recommended, I recommend taking it. It doesn't mean you'll be on it forever, but it will make it a whole hell of a lot easier to work through this to get to the non-medicated point.

</comment><comment><USER14> I can tell you what I know to be true.

Given time, you change and adapt. A great majority of dysfunctions put themselves right if you wait.

If you are patient with yourself you shall be fine. I repeat: be patient with yourself.

Here is my hypothesis or idea of why this sort of thing happens to us.

The brain is this complex organ and when it undergoes stress such as bereavement it shall put you in a certain state, such as low mood, because some part of it not under your conscious control has decided to do this. Seeing this as an error, could be an error. To put it another way, when you injure your leg, you cannot walk on it because of pain. Pain tells you not to hurt yourself more so. The same is true of mental troubles, only instead of pain, it is a lowering of mood. Perhaps this should be understood as a safety mechanism.

In most cases if you run on autopliot for a period of time, perhaps some months or years, you shall recover. In some cases you need to change your environment in some way, but your intuition should be the judge of that.

</comment><comment><USER15> See a psychiatrist, seriously. They'll hook you up with the right combination of counseling and medication (if necessary).

</comment><comment><USER16> >I just work enough get a paycheck, binge watch TV and occasionally go out with friends.

This is a reality millions of people live with (and millions more would love to have). You'll probably get used to it over time.

</comment><comment><USER17> > we were best of friends for 23 years.

Close friends often know our trigger buttons better than we do. If he were here today-- what do you think he would say? What do you think he would do to help pick you up?

There's already several excellent suggestions here. Rigorous exercise is known to help alleviate depression. Honor his memory by making a decision today-- do get moving, start rebuilding.

</comment><comment><USER18> I'm having the same problems right now. I'm 26, entrepeneur and my motivation is going down, way to down. Here is my story: [https://news.ycombinator.com/item?id=12630473](https://news.ycombinator.com/item?id=12630473)

If you feel related and want to chat a bit, let me know! I'd love to chat with someone about this.

</comment><comment><USER19> Sorry for your loss. Take some time to travel out, meditate, enroll for some NGO as helping others without any monetary reason will calm you. we all go through such phase, use this time to find your personal interests.

</comment><comment><USER20> It is quite common for untimely death of someone close to you to cause a depression and it is quite common for that depression to not settle in immediately.

</comment><comment><USER21> Throw out the TV, for starters.</comment>


Show HN: Avoidting – Avoid meetings by appearing busy - <AUTHOR> https://www.avoidting.club/ <comment><USER2> great!</comment>


The Chinese language and the grass mud horse - <AUTHOR> http://irgendwasmitcomputern.de/1/post/2009/03/i-accidentally-the-whole-grass-mud-horse.html

<comment><USER2> Oh 哪个 please!

I disagree with Chinese characters being difficult, at least the part about learning all the characters. There is a methodical approach to learning all the character (there are up to 3000 needed as the article states). I've been using <http://smart.fm/> because they took this approach and wrapped it up in a neat flash app.

So while it takes years to master all the nuances of writing and speaking Chinese, I find that reading and understanding it takes less time.

<comment><USER3> I can read over 3,000 characters and still struggle with reading authentic texts written for adults. Obviously, the language is learnable but the writing system is a lot more work than most languages.

David Moser's essay _Why Chinese is so damn hard_ goes into more detail as to how it is one natural language can be more difficult for foreign students than another.

</comment></comment><comment><USER4> There is incorrect information in the submitted article. It bugs me to read wrong information about Chinese on the Internets because Chinese was my undergraduate field of study.</comment>


Ask HN: What is the value behind OpenSocial? - <AUTHOR>

After reading through Opensocial's site and documentation, I find myself still confused. Perhaps part of it centers on the fact that I wasn't around when it launched.<p>Is it a failed attempt at making apps adopt a universal api? And I say failed because Twitter and Facebook aren't part of the movement. Or is it going to be a force to be reckoned with in the future? Also, how does it fit into the Wave's upcoming launch? <comment><USER2> Facebook might be bigger, but I wouldn't call OpenSocial a failure since around a hundred million people are using it.

I've been writing an app that is supposed to work both on MySpace and Mixi.jp (~20M users), both use OpenSocial. While theoretically you are supposed to be able to write once and run everywhere, in reality you have to tweak your app to handle differences in feed publishing details, checking call signatures etc.

Still it was a lot nicer than having to write the app totally from scratch for each.

<comment><AUTHOR> I don't think it's a failure--obviously other countries are using the social networks that are on open social; however, with facebook in hyper-growth in many countries (primarily south-east asia), this distribution may only decrease in the coming year(s).

Is there any way to combine open social with Facebook?</comment></comment>


Hckrnews.com down? - sid-kap Does anyone know why hckrnews is down? Will it be down permanently? <comment><AUTHOR> 1\. yes 2. don't know, happened before (last week)

same functionality: go to [https://news.ycombinator.com/newest](https://news.ycombinator.com/newest) and paste in console:

for(x of document.getElementsByClassName('score')){ if(x.innerText=="1 point" || x.innerText=="2 points"){ y=x.parentNode.parentNode; y.previousSibling.remove(); y.remove() } }

<comment><USER2> Care to elaborate on the first point?

</comment></comment><comment><USER2> Hopefully it's not permanent. Hckrnews is my main interface for HN anymore.

Does anybody know if the source is out anywhere in case hckrnews.com doesn't come back up?

</comment><comment> sid-kap It's back up</comment>


Show HN: Early version of a Speed Dating Community - <AUTHOR> https://fast-bayou-74365.herokuapp.com <comment><USER3> Folks are getting laid for free via Tinder and others. Why would anyone pay $10/month for a smaller pool of candidates?

<comment><USER4> Maybe not everyone wants to get laid.

<comment><USER5> Maybe, but anyone who would elect to use a "speed dating" app probably does.

<comment><USER6> The point of speed dating is to meet a lot of people quickly in the hope that you'll find someone who has chemistry with you. It's about personality, not sex.

<comment><USER7> Here's my idea for an AI-based speed dating site/app. You fill in the basics (about you and your preferences), plus 5 things that are important to you in life, plus 1-2 things you want to absolutely avoid (ex, smokers), your availability and that's it.

In the background, the AI matches you and sets you up for a quick coffee date at a location that's not far from either person. Twist: every single person actively looking for a relationship in each particular city must be on the site (aside from self-reporting, I haven't figured out how this will be done).

Essentially, you don't peruse the app, you just receive when & where notifications and then go have a 5-minute coffee to see if you have chemistry, which no current app can actually tell you.

<comment><USER8> That leaves you essentially only dating people who are within ~10 minutes of you at some point. Anything more would be a big time sacrifice- a 5 minute coffee turns into 15 minutes with a 5 minute commute. If that person ghosts (as online daters will know happens ALL THE TIME) or is suddenly unavailable, or 5 minutes late or whatever, you're wasting extra time on someone you've never met and might never met.

As an example, I haven't been actively dating in a bit, but when I last was it was in a big city with lots of public transit, and my radius was ~10 miles. If I started talking to someone I liked it was no problem for us to work out a time to meet, but I was constantly interacting with people who were never within "random meetup" distance of me.

<comment><USER6> _If that person ghosts_

To get around the ghosting problem you could use GPS to see if the person is actually where they're supposed to be for the speed date. Essentially, people would get a notification when >x people are available in the area and they could go to a coffee shop and flip a toggle which instructs the app to start sending people to speed date them. If they leave the coffee shop or turn off their phone or anything else then the app would toggle off their availability and stop sending dates.

It could be similar to the way Uber manages driver availability in an area. Except instead of sending you fares, it sends dates.

The beauty of a system like this is that you could go to a coffee shop and read or do work on your laptop and toggle on your availability. If people don't show up to date you it doesn't matter because you're not wasting time anyway.

<comment><USER7> Nice pivot.

I'm imagining a toggle to alter your visibility too, between showing your general location within a venue, to something more playful (I'm wearing a green shirt, when there are two others in the room with green shirts), or even just 100% visible (here's my picture, I'm sitting exactly here).

Though, I'm now reminded of two r/dataisbeautiful posts [1,2], where - <USER2> both having faulty graphs - the OP shows how s/he got the most dates when the women messaged first. Meaning, this imaginary app we're discussing might have a ton of guys bombarding female users, if it weren't for the AI being involved, as it will be the one matching nearby suitors for you. A search of 'dating' in the subreddit shows a lot more results.

1 - [https://www.reddit.com/r/dataisbeautiful/comments/7oqygb/my_...](https://www.reddit.com/r/dataisbeautiful/comments/7oqygb/my_500_days_on_okcupid_oc/)

2 - [https://www.reddit.com/r/dataisbeautiful/comments/7p73ch/my_...](https://www.reddit.com/r/dataisbeautiful/comments/7p73ch/my_180_days_of_lesbian_online_dating_oc/)

<comment><USER6> _might have a ton of guys bombarding female users_

Then perhaps the solution is to make guys be the ones to sit in the coffee shop and have the women decide who they want to drop in on.

The really beautiful thing about this is that the woman could see the guy and decide whether she thought it was safe to approach him or not without him even knowing she's there. She could just use the bathroom or buy a coffee and leave or anything else, just like any other customer.

<comment><USER7> Although this is why I continued that statement with "if it weren't for the AI being involved, as it will be the one matching nearby suitors for you."

Meaning no bombardment will be allowed cause the users (neither male nor female) won't be the ones sending first contact.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> I want to login with my own email and pass. Login with FB and/or Linked In is not an option for me.

<comment><AUTHOR> I dislike social profiles like the next person and would like a better way of verifying...

I just had an idea whilst thinking about how I was going to respond to this -- perhaps I could do e-mail signup but instead verify the user through the clearbit/fullcontact api. Need to look into that

<comment><USER10> How is a Facebook or LinkedIn account considered "verification" when spammers routinely use thousands of fake accounts to astroturf fraudulent pages & ads? Surely this means any malicious actor targeting your platform can do the same.

</comment></comment></comment><comment><USER11> The problem with dating isn't lack of meeting people, it's lack of heuristics. Speed dating just makes it faster to judge, but it doesn't actually do any work for you. One of the big benefits to OkCupid is that the heuristics are relatively good. Improve on that, and you're golden.

<comment><USER12> one of the problems with good heuristics is that almost all the desire is sent towards the 9's and 10's. Yet obviously most of the site users are 8 or less. So the 9-10s gets swamped, and leave early. The remaining users gets bored because nobody replies.

The problem here is not that heuristics of finding "your best match" is bad. It is quite easy to figure out who is attractive and matches you. The problem is that most people wants to trade up, and few people (on dating sites) have the desire to end up with someone their own score. Fewer yet wants to date someone less attractive. Hence the dating market ends up being a sort mechanism which works poorly because none of the sorted numbers wants to be where they are.

OkCupid had some great blog articles about the statistics behind this unbalanced matching phenomenon. Read them.

<comment><USER13> Not the dating market, just the online dating market

People date up and down all the time, and attraction isn't just limited to good looks. People are attracted to different things in an individual (good looks, ability, wealth, intellect, power or status, etc) -- it's just that those don't always show well on dating profiles. They are much easier to spot in the real world.

</comment></comment></comment><comment><USER14> Interesting concept. How do you aim to verify people?

<comment><AUTHOR> Currently, * unless someone has a better idea * it would be through manually checking any social profiles including git profiles. The aim is not to turn it into a 'club' but just be selective enough that there are alot of like-minded individuals.

</comment></comment><comment><USER15> A comment on design: I'd recommend removing the large buttons for "How it Works" and "Queue up" on the landing image and instead using some visual indicator to scroll down. Scrolling down vs. clicking buttons (that just automatically scroll you down) seems redundant to me.

<comment><USER16> I saw this trend recently on various websites. Maybe it's born out of (unjustified?) fear that people don't scroll below the fold?

<comment><AUTHOR> Possibly unjustified fear, that is why the current website has it at the moment...

</comment></comment></comment><comment><USER17> I really like this concept OP. I think your biggest non-technical hurdle is going to be dealing with the classic issue of there simply being significantly less women than men. I'm curious to know how you plan to approach that after you introduce a proper business model.

<comment><USER18> Former speed dating entrepreneur here. In person, at least, women are much more open to the idea of speed dating than men. Much of our marketing revolved around how to appeal to and attract men to our events.

<comment><USER17> Interesting, I'm curious to know what kind of speeddating events/services/apps are out there. I've never used any.

</comment></comment></comment><comment><USER19> Sounds a bit like chat-roulette.

<comment><USER20> Yeah, the real effort is this site not changing into that.

<comment><USER19> The problem is that it can become insincere very quickly.

Like when a group of people decides to have some fun by laughing at other people who are hoping for a date. This could happen even on a "genuine" account, because intentions of people can change. And people don't act the same when alone versus with friends.

<comment><USER21> Or pervs getting on naked, and totally not expecting it...

</comment></comment></comment></comment><comment><USER22> How about a company that facilitates physical speed dating meetings? That'd target an older demographic.

</comment><comment><USER23> The queue up button does nothing on my phone. iPhone x, 11.4 in Safari

<comment><AUTHOR> just fixed it, thanks for that!

</comment></comment><comment><USER24> Looks very cool! :)

<comment><AUTHOR> Much appreciated, thanks alot

</comment></comment><comment><USER25> queue up does not work

<comment><AUTHOR> just fixed it, thanks for that!

</comment></comment><comment><USER26> it's not going to fly. I think the problem of this kind of ideas is that dating is actually a by product of social interactions, not suitable for engineering.

<comment><USER27> It may not fly or it may rocket! Either way, encouragement costs nothing. I wish them all the success. Good luck!

<comment><AUTHOR> Thanks!</comment></comment></comment>


TrustFabric - Most ambitious startup ever - <AUTHOR> http://tylerreed.com/startups/trustfabric-most-ambitious-startup-ever.html

<comment><USER2> Starting a private space organisation is ambitious. Starting a new operating system business is ambitious. Starting a CRM variant in which the customers manage their details for you is... Ambitious, I suppose, but the title of this submission is definitely hyperbolic.

<comment><USER3> There's Tracey Davis ambitious, there's Draco Malfoy ambitious, and then there's Harry James Potter-Evans-Verres ambitious. The top end of the scale isn't "start an new OS business", it's things like molecular nanotechnology or self-improving artificial intelligence.

<comment><USER4> Speaking of Harry James Potter-Evans-Verres, come on! It's been almost two months! I need my HPMoR fix ;-)

<comment><USER5> Am I the only person who feels a bit sorry/annoyed every time Eliezer gets harassed about updating HPMOR? It's his project, people, he'll update when he wants to.

<comment><USER4> It was supposed to be totally tongue in cheek with a wink and smiley to make it obvious. I suppose that is still harassment. I am sorry.

</comment></comment></comment></comment></comment><comment><USER6> This requires me not just to absolutely trust trustfabric, but to trust that they're security is bulletproof and that their identity reclamation mechanisms such carefully engineered paragons defence in depth.

Given their admittedly very clear privacy policy does not rule out selling your information in aggregate or in detail nor changing the policy without explicit notification they've fallen before the first hurdle.

We desperately need a neutral provider of cryptographically secure identity, ideally that can be verified against 3rd parties such as banks or passports. The problem is I don't know what the business model for that looks like.

<comment><USER7> I'm not sure which privacy policy you're reading, but the one on <https://www.trustfabric.com/connect/privacy-policy/> makes it pretty clear that they'll never sell your information.

<comment><USER6> That was not there when I wrote that. See google cache. [http://webcache.googleusercontent.com/search?q=cache:Kyue2dB...](http://webcache.googleusercontent.com/search?q=cache:Kyue2dBfpNMJ:https://www.trustfabric.com/connect/privacy- policy/+https://www.trustfabric.com/connect/privacy- policy/&cd=1&hl=en&ct=clnk&gl=uk&source=www.google.co.uk)

</comment></comment></comment><comment><USER8> Sit down 100 randoms from off the street, ask them if "companies having your data up to date" is a big problem for them. The ask them how high up their list this problem is, if of course it actually is.

My money is on "not very high".

<comment><USER9> I think this is a bit too cynical.

If I understand this right - not only do you avoid having to fill out the same address forms over and over for different services, you avoid the monumental task of making sure all of your information is up to date on major life events like moving, marriage, etc.

If I could avoid typing my shipping addresses in every online order form sourced from a trusted place (browser auto-fill is shit), I'd use the service in a heartbeat. Only updating one service when I move apartments? Sign me up.

<comment><USER10> If I could avoid having to fill out twelve forms - nearly identical for the first half of the page - every time I went to the doctor, that'd be worth $20 bucks a year from me.

But I'm pretty sure that developing a HIPAA-compliant system, and then convincing every doctor's office in the country to switch to it won't be covered by a $20/year subscription fee.

</comment></comment></comment><comment><USER11> I think Thawte was a much more ambitious startup to come out of South Africa.

Thawte was founded in 1995 by Mark Shuttleworth in South Africa and was originally run from Shuttleworth's parents' garage. It became the second largest public certificate authority on the Internet and sold for hundreds of millions of USD within a few years.

<comment><USER12> I disagree, and you will notice lots of skepticism in the comments on the this thread about the idea itself. The reason the idea is more ambitious is because it requires a change in thinking and behavior on the part of the consumer. I was part of a company trying to start a VRM company 3 years ago and it was very difficult to get the idea across to possible investors.

The reason it was so difficult to explain was because the starting premise centered around technology that still isn't in the market: Near Field Communication (NFC). As NFC becomes a physicalID<\--->digitalID, and serves as a basis for payment, the companies involved in payment processing, the banks, and the merchants one patronizes become quite capable of compiling vast amounts of data that could be seen as quite personal. However, because the concept seems very convenient in many ways, those that are uncomfortable with the potential data mining will still be using cash. So, the idea was to put a buffer between the people and the businesses that had yet to earn one's trust.

</comment></comment><comment><USER13> Before TF can get users on board they must first get businesses on board. In order to accomplish the latter they must ask businesses to create momentum (which doesn't currently exist) that would, over the course of time if successful, reduce their ability to employ current marketing strategies towards their customers.

Yes, I agree that from a business owner's perspective dollars spent marketing towards prospective customers who want to hear from you are more efficient outflows than towards those who do not want to hear from you. However, one underlying concept of marketing to new prospective customers is the introduction of a novel product for which they previously were unaware existed. Therefore, how would a customer who does not know that a product produced by Seller X existed approve Seller X to begin soliciting their business? This may be a fundamental flaw in the business or maybe the YouTube video is just more customer-centric than business to make this apparent to potential business customers (for which I assume are the ones who will generate revenue). I'd like to see their pitch to business customers before deciding either way. I like the concept but I am not yet sold on some of the inherent biz model fundamentals as of yet. I'm sure that over time they will work this and other issues out.

In most cases first-to-market provides an advantage. However, in some instances of extreme novel concepts where timing isn't yet appropriate, first- to-market simply provides paving of a runway for others to take flight from and eventually leap-frog the trailblazers whose time & resources made things easier for those before them. This concept may be a bit early (aprox 5 years) but I wish them luck as I am always pulling for the Pioneer rather than the better-timed copycat.

</comment><comment><USER14> "Most ambitious startup ever" ...in South Africa, the article continues.

</comment><comment><USER15> I've kicked around this idea on and off for a number of years. I first remember having this in 2004, because I started to move - we moved 3 times in two years, between 3 states. Repeating the same stuff, registering for the same services, etc. all got rather tedious. By 2007/2008, this idea _felt_ like it would dovetail nicely with openid, but that subsequently failed to take off.

Was it because of the decentralization aspect? Techies seem to be in love with 'decentralization' and 'federation' (witness the diaspora love last year), but average people aren't, and you need average people to be using a system like this in large quantities before companies who might use it _have_ to take you seriously.

This scenario they are tackling is _classic_ chicken and egg, and I'm not sure it's going to be solved outside of a major player like google, facebook or microsoft (or _potentially_ LinkedIn, Apple or Yahoo, but both are longshots). The population at large is starting to grasp the concept of an online identity, as well as the benefits and challenges of controlling aspects of privacy over the web. I don't want to say it would be trivial, but certainly doable for FB or Google to give you a way to allow ID pulls from companies wanting to sync your data in to their CRM systems - they're already doing it with loads of games/apps. A bit more granularity and a nicer UI would go a long way in this space to not only owning and defining it, but preventing any serious competition from upstarts like trustfabric.

If TF can find a way of leveraging existing SN identities as a jumpstart, they may have something. But... for mass adoption, I fear the brand recognition is likely going to keep us tied to a FB/GOOG duopoly for a while.

</comment><comment><USER16> Without a doubt, not writing my address every again would be a delightful state of affairs. However, this seems like it has the potential to make existing online identity theft much, much more serious.

Losing your e-mail address currently requires hard work on the apart of the assailants to cause you actual financial harm. With this they will be able to interact with real world vendors as you.

On the plus side, exciting times ahead! This is the sort of thing that reminds me we are living in the future. Good luck launching.

</comment><comment><USER17> Bad idea. I don't like to even fill out my census form and they would like to sync my info with corps? No. There's more to it that trust, and overall it doesn't seem to fix any need I have. For the general population, I see one immediate benefit: for people on the move, managing change of address and keeping their mail flowing to the proper place is important. It seems that the post-office can do that for you though at a small premium and re-direct every piece of mail. So what exactly does this offer that we need?

<comment><USER12> I completely agree with your trust concerns, but it doesn't seem you are familiar with VRM [1]. The whole point is to off-set the current (and ever growing) data disparity between consumers and businesses. You'll note they have trust in their name, which I would guess is specifically because they realize the significant trust level they're going to have to engender among their customers. I still believe in VRM as a concept, but it has a large barrier to entry mostly because it isn't an easy idea to grok. It's still a vitamin, not a pain pill, but I believe it will be a pain pill eventually.

[1]<http://cyber.law.harvard.edu/projectvrm/Main_Page>

</comment></comment><comment><USER18> Server has not been resized... hopefully it holds up now. Went from 512MB to 4096MB, I should tweak the configs a bit more. That's what I get for being lazy.

<comment><USER19> I find it quite puzzling that "cloud computing" is still so manual and (virtual) "server"-centric. Bumping up numbers in config is less work than gutting a rack to be sure, but surely there is room to apply Apple's hands- free design philosophies to cloud infrastructure?

Food for thought.

<comment><USER20> There's certainly systems out there built to scale automatically. I was on a small team that wrote such a system for an ad network.

We tracked clicks on our own (physical) servers but for impressions, you're talking a system built to serve upwards of 30bn a month. While nothing compared to industry-leaders (Google, DoubleClick, etc), that's still around 10,000 impressions a second.'

We had no problem building a system on our inhouse servers to handle that 10k/s number: A very thin PHP instance running on port 80 that does nothing but push the impression into a Gearman queue. Easy enough, even at that scale.

The problem of course is that you're not getting 10k/s. Some seconds you may get 100k, others 1k.

So we built a simple system that automatically spawned S3 instances when we began to see high loads, distributed the latest code to them, and when we no longer needed them, we downloaded the DB they held locally and moved the data to HDFS.

What we did is pretty common, I'm sure. It's all pretty straightforward software.

</comment></comment></comment><comment><USER21> I really like the idea of being able to centralise all of my information and be able to control who has access to certain sections of it all from one dashboard.

Facebook has gone some of the way to doing this, with me being able to remove access to apps I've previously given access to; however, I'd like to be able to upload customize custom documents, such as my passport or proof of residence, and then be able to share them with specific companies.

It seems like TrustFabric may go some of the to enabling the above?

</comment><comment><USER18> Sorry, never expected the post to reach the front page of HN. I'm busy bumping up my Rackspace cloud server to handle the traffic.

<comment><USER22> Out of curiosity, about how many hits per minute is the #1 spot sending you?

</comment></comment><comment><USER23> Just out of interest, but how big do you estimate the South African market size to be for this? And if looking at it globally, would Americans be will to trust personal data to a South African business or any business for that manner? Especially in light of the recent coups by Anon and Lulz? Btw South African too!

<comment><USER20> I'm a bit of a Mercedes enthusiast. They began making cars in SA some years back. I'm always surprised at the number of owners I talk to who actively seek to avoid the South African built cars. It's possible to do because each model has at least _some_ German production, but it often means a special order and waiting months for the car to sail over.

People are strange. I think a lot of Americans especially do not really know how to categorize South Africa.

</comment></comment><comment><USER24> This is exactly the kind of things that the Obama Administration's NSTIC policy aims to create. I feel like a number of companies (Google, Facebook, your bank, specifically) already have plans to do this.

<comment><USER12> I guess my entire interest in VRM is because I do not like being data mined, and you just named _the_ top candidates for data mining that concern me.

EDIT: When the NSTIC was announced, many organisations I respect, and typically agree with, game out against it[1]:

 _"Shortly after the draft's release, the Electronic Privacy Information Center, in conjunction with a number of other consumer-rights and civil liberties organizations[5], sent the Committee a statement in response to the draft NSTIC policy, requesting that the White House provide a clearer and more complete plan to create and safeguard Internet users' rights and privacy."_

[1]<http://en.wikipedia.org/wiki/NSTIC#Criticism>

</comment></comment><comment><USER7> Blog is taking strain, you can view their video here <https://www.trustfabric.com/connect/>

</comment><comment><USER25> Can anyone give us a summary of this ambitious start-up?

<comment><USER25> I am not sure what the link is to but their site is up, <http://www.trustfabric.com>

"TrustFabric is a startup based in Cape Town, South Africa, writing Vendor Relationship Management software. VRM gives customers a platform to represent their side of the VRM+CRM relationship. TrustFabric gives individuals control over their information.

The TrustFabric platform allows individuals to keep their personal information up-to-date in one place and then selectively and securely share that information with the organisations they have relationships with."

</comment></comment><comment><USER26> I wonder if they implemented it in a way they never share your real details with any business, proxying everything through their offices?

</comment><comment><USER27> ...to come out of South Africa.

</comment><comment><USER28> Amandla!</comment>


Ask HN: How much would you give to a non-profit to rewrite emacs in scheme? - <AUTHOR>

I think part of the reason Guile failed is that it just sort of petered out. I think any attempt to replace elisp with scheme will have to start with a bang to overcome network effects. The hypothetical nonprofit would wait until they had enough money to do so, and then start development. (They could hold the money and any interest in an escrow account to reassure donors.)
Personally, I hate dynamic scoping and having 2 namespaces. Also, whenever I see a piece of emacs is written in C I think it's best to just use the C function as written and not even try to understand it. Having scheme all the way down would dramatically lower the psychological barrier to fiddling with low-level functions like font-rendering. (i.e. I don't think it would have taken till version 23 to get anti-aliased fonts if it was scheme all the way down)
What do you all think? Bad idea? Good idea? Do you also hate lisp-2s and dynamic scooping? <comment><USER2> Climacs and Hemlock have existed for many years, but there's been no flood of users. Both are high-quality emacs-like editors written in common lisp; both have many extensions for development, mail, news etc.

I can't imagine a donation-supported project going from scratch would be substantially more successful just because it used Scheme instead of CL.

Climacs: <http://common-lisp.net/project/climacs/>

CMUCL Hemlock: <http://www.cons.org/cmucl/hemlock/>

Portable Hemlock: <http://common-lisp.net/project/phemlock/>

P.S.

I agree with your goals: I really don't enjoy elisp at all. It reminds me just enough of CL and Scheme to be sick at what it lacks.

<comment><AUTHOR> Edit: Maybe it makes sense to hire a marketer or five? Also, lots of eye candy.

1\. The longer we wait the more elisp is gonna be out there and the more painful continuing to maintain existing elisp, and the eventual transition will be. Nobody wants to be the one to muck through it all(for free), but the payoff if the project is successful is so huge I think the risk-to-reward ratio makes sense. Also, rewriting emacs would beat the hell out of most coding jobs out there.

2\. Scheme was made to be as minimalist a language as possible, making it easier to wrap your head around the whole language. And more importantly, lowering the learning curve a little. (I've been bit in the ass by inconsistencies in how elisp handles things before.)

3\. SICP is in scheme. I think it'd be great for the language people learn 'hello world' in to be very similar or identical the language they mod. their text editor in. A single smooth experience, instead of the jarring shock when you bump against elisps rough edges.

4.Again, lisp-1. I believe I could've spent lot less time writing in emacs- lisp if it had a single namespace. It certainly makes it easier to have code- as-data.

<comment><USER2> Your reply seems to discuss elisp, not Common Lisp. elisp is not CL. Scheme and CL have more in common than elisp and CL or elisp and Scheme. (OO/MOP, namespacing, modern compilers, lexical scope)

My assertion was that there are already very sophisticated alternatives to emacs, written in a real, modern language.

elisp is terrible, but the failure of CL-based emacs makes it clear that popularity is more important than ease of use or ease in learning. I don't see how the use of Scheme would make a difference.</comment></comment></comment>


Download.com Response to Nmap Offer Bundling - <AUTHOR> http://download.cnet.com/8301-2007_4-57338809-12/a-note-from-sean-regarding-the-download.com-installer/

<comment><USER2> BTW: CNET is not the only entity doing that, they just happen to do it on a scale and got noticed by the public.

There's a class of lowlifes who take popular open source programs, rename it and wrap inside a custom installer that similarly tries to trick the user to install some other, usually crappy, software.

My program (Sumatra PDF) is popular enough that it happened to me 3 times and I know that other popular open-source programs were similarly violated by the same people (not going to link to them or even mention names to not give them google juice).

<comment><USER3> I love Sumatra PDF, thank you for making it.

</comment></comment><comment> anon-for-this Email to developers from download.com:

Download.com Developer Community,

My last communication to you was shortly after we launched the Download.com Installer in late summer. At that time I asked for patience as we began work to deliver a mutually beneficial model to market.

We are on the verge of fulfilling our vision of coming to market with an installer model that delivers files faster and more efficiently to users, while enabling developers to a) opt-in to the Installer, b) influence the offers tied to their files, c) gain reporting insight into the download funnel, and d) share in the revenue generated by the installer. However, due to some press that surfaced yesterday and the potential for subsequent misinformation, I am reaching out now to address that press and to provide a progress report on the upcoming launch:

First, on the press that surfaced yesterday: a developer expressed anger and frustration about our current model and how his file was being bundled. This was a mistake on our part and we apologize to the developer and user communities for the unrest it caused. As a rule, we do not bundle open source software and in addition to taking this developers file out of the installer flow, we have gone in and re-checked all open source files in our catalog. We take feedback from our developer & user communities very seriously and take pains to both act on it and respond in a timely manner.

With that, I want to share progress made thus far: This week we will launch the alpha phase of our new installer. This alpha phase is intended to test the tech and do QA, and will roll through the next few weeks to ensure that our installer is bug free. Between this week and the end of January we will be completing the necessary engineering and administrative work to roll out our beta, which will include a small group of developers who've agreed to participate in the beta launch. Our goal is to exit beta by end of February and have the necessary systems in place to enable opt-in, influence over advertising offers (for those offers that impact your product), download funnel reporting and revenue share back to you, the developers. In the weeks/months following the full release, we will continue to iterate on the model, adding more features to the Installer and bringing greater efficiency to our own download funnel (read: increased install conversion). The initial feedback from developers on our new model has been very positive and we are excited to bring this to the broader community as soon as possible. More communication will follow as we move into Q1, and until then, thank you for continuing to work with Download.com.

Sincerely,

...

</comment><comment><USER2> They are lying.

I just checked and my very much open-source Sumatra PDF is still wrapped in their cnet installer which tries to push some "special offers" or trick the user into installing some other software.

"In addition to immediately taking Nmap out of the download manager, we reviewed all open source files in our catalog to ensure none are being bundled."

Like hell you did, CNET.

<comment><USER4> If you read the comments, you'll see they also missed putty and filezilla. Their 'review' indeed seems to have been pretty poor.

<comment><USER2> Yeah, I noticed. I also asked them to remove bundle from my program via comments. It doesn't seem like there's any other, documented, way to get them to do it.

<comment><USER5> IANAL; Can you legally threat them for integrating closed source with your GNU GPL?

Might also be a good idea to warn people via your download page. No one knows where else to get it.

</comment></comment></comment></comment><comment><USER6> This response doesn't respond to the claim that download.com is bundling other software as well (the original nmap post referred specifically to VLC). This response claims that wrapping nmap was a "mistake", should we conclude that wrapping other software was therefore deliberate?

<comment><USER7> Of course wrapping the other software was deliberate! Someone at c|net designed, built, and deployed the wrapping feature; that is as deliberate as you can get.

However, _at least_ they have re-enabled the direct download link for non- registered users so that those people who understand the difference can avoid the wrapper. I still think it's distasteful to pray on the ignorance of users, as all these wrappers and toolbars do, but that is how they get paid.

</comment></comment><comment><USER8> I like how they apologized for the _unrest_. That is, they apologized for how people reacted, not what they did.

<comment><USER9> That's how you know it's a company run by bad people.

<comment><USER10> I think it is instead a bad company that runs people. Just because C/Net tends to attract morally ambiguous people who have no qualms twisting the facts to suit their marketing/business campaigns is a side effect of their mission statement, "As leading destinations for the information and entertainment people crave(1), we don't just support lifestyles – we help define them(2)." via <http://www.cbsinteractive.com/company_info.php>

NOTE: They are in the CBS Interactive brand portfolio.

(1) Meaning media=software downloads for Downloads.com

(2) Through bundling malware which customers wouldn't intentionally install otherwise.

<comment><USER9> It's the result of actual people making decisions about what to say.

</comment></comment></comment></comment><comment><USER11> So are they saying they only intended to bundle the crapware with closed source software, or am I misunderstanding?

<comment><USER12> That's what I took away from their release too. I'm guessing they know some open source licenses specifically forbid what they did due to the nmap posting and are just trying to CYA.

<comment><USER13> Most open source licenses don't forbid 'mere bundling' with some other software, even if said other software is Evil. Most likely download.com just doesn't want to have to deal with source distribution requirements from GPL/LGPL software.

<comment><USER12> I don't doubt that you are correct. This is what I was recalling and referring to from the seclist post regarding nmap:

This is exactly why Nmap isn't under the plain GPL. Our license (<http://nmap.org/book/man-legal.html>) specifically adds a clause forbidding software which "integrates/includes/aggregates Nmap into a proprietary executable installer" unless that software itself conforms to various GPL requirements (this proprietary C|Net download.com software and the toolbar don't).

Can't imagine they would want to release the source code to their software at any point, it's easier to just make a blanket statement that you don't intend to bundle any open source software with it. Even if that's not what you actually do which seems to be the case according to some of the other comments.

</comment></comment></comment></comment><comment><USER14> Translation: We go through a rigorous process to ensure that we don't give you any spyware / adware / malware...

...except, of course, our own.

</comment><comment><USER15> Quote: _It is a Download.com policy not to bundle open source software [...]_

To the best of my knowledge, no popular FOSS license forbids bundling. But if they bundled them, they'd have to deal with distribution of the source code too (which is a PITA for so many products of so many different versions and licenses), but possibly they'd also have to open source their installer (can't see it being such a big trade secret, but CNet's that kind of company where they'd all go OMG! Competitors steal our code and ideas!) which is probably why they're refraining from doing so.

Now it's interesting that, reading between the lines, they'll continue to do this for free, non-opensource software (like mine)..... which ironically _can_ and oftentimes _do_ (I do) explicitly state that commercial use is restricted and bundling isn't allowed.

I'd say this response is jumping out of the frying pan (technical issues and obstacles) and into the fire (illegal, in direct violation).

<comment><USER16> The makers of EasyBCD (non-FOSS freeware) have become pissed about this, and demanded that download.com unbundle it, or delist it because it is a violation of the license.

Throughout their letter they talk about how them taking action was only possible because of their proprietary license.

</comment></comment><comment><USER17> Anyone know how to remove software from being listed on download.com? In My Products on upload.com I don't see any options, nor anything in the FAQ. I emailed them, but I suspect it will be days before I get a response...

<comment><USER17> (I would have edited my response to add this to my original reply, but there is no "edit" link for it)

Apparently you have to email them and request removal and you need to give a reason. I'm not sure if they would consider a reason invalid or not. We'll see. I've requested removal of 2 products from their site.

</comment></comment><comment><USER18> They also wrap AbiWord (GPLv2 licensed) in the cnet downloader..... that ask you if you want to install a toolbar. This does not look good either.

Time to lawyer up a bit.</comment>


Flattr now deletes your web browsing history within 3 months - <AUTHOR> https://www.ctrl.blog/entry/flattr-gdpr <comment><USER2> I love how some of the tech industry is beginning to see data as a liability rather than an asset. It dramatically reduces the ability for government mass surveillance for two reasons:

1\. If companies only collect what they need (to reduce their liability), governments can't demand more than that (or even hack in to get the data illegally).

2\. If the industry culture is to limit data collection, governments can't just say, "Well every company does it, so why can't we."

There's a wonderful talk, Haunted By Data, that covers a lot of the societal downsides of treating data as an asset. Highly encourage watching/reading.

Text: [http://idlewords.com/talks/haunted_by_data.htm](http://idlewords.com/talks/haunted_by_data.htm)

Video: [https://www.youtube.com/watch?v=GAXLHM-1Psk](https://www.youtube.com/watch?v=GAXLHM-1Psk)

<comment><USER3> Just to be contrarian: I'm not so sure if that's a good thing... Data can be used for great things, e.g. longitudinal data in healthcare. I think seeing data as a liability might reduce the speed of progress

<comment><USER2> In the talk, there's a parallel drawn between the nuclear industry 60 years ago and big data now, where the nuclear was originally touted as a miracle cure for everything, then disasters happened, then it never really got over the stigma despite its huge potential.

Society decided, for now, the upsides aren't worth the downsides.

Oil is another parallel where society is currently just at the point where we are starting to not value the upsides over the downsides.

<comment><USER4> Seems like oil and nuclear energy are different because both can be replaced with alternative sources of energy. What are the alternative sources of user data?

<comment><USER5> Paper records, human memory....

Comparing growth in data storage versus energy usage per capita is interesting.

Even if you look back to the founding of the U.S., the change in energy use per person is actually only a few fold, definitely less than an order of magnitude.

Harder to compare quantity of data storage but the change would seem much larger. How much data is there, per U.S. person?

</comment></comment></comment></comment></comment><comment><USER6> I feel like this whole thing with the GDPR is people fighting each other for the wrong reasons.

The people for it argue that gets large companies to behave better.

The people against it argue that it's unnecessarily complicated, poorly drafted and burdensome to small businesses.

But it's _both at the same time_.

The issue is that it doesn't have to be. It's possible to get the desired effect without using such a complicated system, and to mitigate the impact it imposes.

For example, one of the major burdens on small businesses is the cost of producing the data the company has on you -- but there is a simple way to fix that. Require the requester to pay the cost of collecting the information, similar to FOIA requests in the US. Then the requirement is no longer an unfunded mandate and can't be used by griefers as a method of harassment.

Nobody bothered to use common sense like that when drafting it, so now in order to fix it, it has to be thrown out and rewritten. But then the people who support the spirit of the law end up fighting against the people who oppose the letter of the law, even though everybody really wants the same thing.

<comment><USER7> "cost of producing the data" seems really hard to strictly define. What's stopping a company from implementing extremely tedious procedures and billing $100 an hour to discourage people from requesting it?

In addition, GPDR says that getting access to data on you is a right and it seems contradictory to have to pay for something that is your right to have.

<comment><USER6> > "cost of producing the data" seems really hard to strictly define. What's > stopping a company from implementing extremely tedious procedures and > billing $100 an hour to discourage people from requesting it?

Government agencies have been known to do this when they don't want to release FIOA information. Then they get sued over it by the ACLU or similar.

> In addition, GPDR says that getting access to data on you is a right and it > seems contradictory to have to pay for something that is your right to have.

That is how all the other rights work. Freedom of the press isn't a right to use someone else's printing press without compensating them.

<comment><USER8>
       access to data on you is a right
       Freedom of the press isn't a right

Hence the classic liberal difference between rights to something and freedom to do something

<comment><USER6> By this definition all of the usual "rights" are really freedoms. Which leaves me waiting for an explanation of why accessing data a third party has on you should be elevated above the freedom of speech, travel, association and so on as something private parties should be required to do for all comers without compensation.

</comment></comment></comment></comment></comment><comment><USER9> "Ads on this site don't track or stalk you. Please disable your blocker."

According to my blocker those are Google ads so...

<comment><AUTHOR> Google AdSense on Ctrl blog is configured to only show non-personalized ads to any users with the Do-Not-Track (DNT) setting enabled or European Economic Area (EEA) citizens. AdSense still uses cookies for rate-limiting and fraud prevention, but not ad personalization or tracking. requestNonPersonalizedAds=1 is part of AdSense's GDPR APIs. [https://www.ctrl.blog/entry/adsense-gdpr- consent](https://www.ctrl.blog/entry/adsense-gdpr-consent)

You also only see that particular message if your browser sends the DNT header, and an adblocker is detected.

<comment><USER10> In my case, I don't use an ad blocker, but I do use Privacy Badger, which is only supposed to block tracking, and should only block ads if they can't have their tracking disabled. It looks like Privacy Badger decided "pagead2.googlesyndication.com" should now be blocked as a tracking domain.

Perhaps Google should split off their nonpersonalized ads service into a different domain so such services can easily distinguish between the two?

<comment><AUTHOR> Actually, this is something that should be fixed in Privacy Badger. They can detect the URL parameter that flips ads from personalized to non-personalized (npa=1) and block accordingly. [https://github.com/EFForg/privacybadger/issues/2046](https://github.com/EFForg/privacybadger/issues/2046)

<comment><USER11> That doesn't make any sense. The point of privacy badger is to prevent cookies that track you across different domains.

Allowing tracking cookies just because a certain parameter is present is asking for abuse.

If you are serious about not tracking people, don't use tracking cookies.

<comment><AUTHOR> Cookies is just a tool it all comes down to how you use it. Not all knifes are stabbing-people-to-death knifes.

<comment><USER11> Privacy badger is quite specific in what it blocks. It blocks third party tracking cookies that track you across multiple domains.

First party cookies are no problem. Third party cookies that are only used in one domain are no problem. Third party non-tracking cookies are no problem.

So it does block the equivalent of bringing a large knife to a bar.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> > install the company's browser extension which collects their browsing > history

Yeah I think I'll pass thanks. What person would install such a thing?

<comment><AUTHOR> Well, … people choose to install and use Google Chrome … It all comes down to who you trust with your data.

<comment><USER12> It's hard to use the internet or live without a web browser, I suspect most people get along just fine without letting a startup with a cool name scrape their browsing history.

</comment></comment></comment><comment><USER13> Wtf?! Why is micro payment processor even recording browser history?!

<comment><USER14> From the article:

"Flattr subscribers make a voluntary payment from 3 USD/month, install the company's browser extension which collects their browsing history, and then Flattr divides their subscription fee out among the creators and websites they spent the most time on."

<comment><USER15> That could be done in a totally anonymous way. Add up total number of donations and total number of visits and distribute money accordingly. There is no need for them to ever store personal information for this.

<comment><USER16> (a) How do you get the total number of visits.

(b) This doesn't correctly distribute funds because there is a correlation between how much someone is willing to give per month and what kind of sites they frequent.

<comment><USER15> You can calculate the numbers as soon as the data about visits come in and then immediately discard all data. You certainly don't need to store the browsing history. You need the payment amount though and even that you can discard after a month. I am going through this at work right now where we are working data collection about a medical device. First we wanted to suck up all data and figure out what to do with it later. Now we have to think about the data use upfront and get that data in the least intrusive way. I requires more thinking but we get the same benefit with only a fraction of the data. GDPR is a good thing because it causes companies to rethink their data collection strategies.

<comment><USER17> There isn't enough data to calculate numbers as soon as the visits come in. If I pay $3/month and visit your site, the amount you get depends on how many other site visits I accumulate during the month. If I visit 2 sites, you get $1.50. If I visit 20 sites, you get $0.15.

You also need to store browsing history to avoid scammers redirecting other users donations. This happens on similar platforms like Spotify. You register for Flattr and just run a script to constantly visit your site. With aggregated statistics, this shifts a bit of everyone's donations to you and can be extremely lucrative. With session history, you just get your own donation back, minus fees.

<comment><USER16> Thank you. The scammer aspect is exactly what I was getting at. Without allocating any given user's donations fairly across the sites they have individually visited, I don't see how it works. It'd be susceptible to massive click-fraud.

</comment></comment></comment></comment></comment></comment></comment><comment><USER18> Except its not 'in practice' because gdpr does not require such thing. In spirit, maybe

[i.e. _practically_ they wouldn't be fined for this]

<comment><USER19> It might be reasonable to conclude that, in practice, GDPR has caused a number of companies to their re-assess data collection and retention hygiene (even beyond the minimum bounds of the law). In particular, this change seems to be a very charitable or expansive reading of the requirement under the GDPR for companies not to collect more than is necessary -- as the post ends by noting.

<comment><USER20> I think hygiene was mostly in place, but there was no PR points to score. Before GDPR an information that company stores something for 3 months would be a non-news. GDPR doesn't in any way protect people from data leaks.

<comment><USER21> >GDPR doesn't in any way protect people from data leaks.

Article 32:

 _Security of processing

Taking into account the state of the art, the costs of implementation and the nature, scope, context and purposes of processing as well as the risk of varying likelihood and severity for the rights and freedoms of natural persons, the controller and the processor shall implement appropriate technical and organisational measures to ensure a level of security appropriate to the risk, including inter alia as appropriate:

(a) the pseudonymisation and encryption of personal data; (b) the ability to ensure the ongoing confidentiality, integrity, availability and resilience of processing systems and services; (c) the ability to restore the availability and access to personal data in a timely manner in the event of a physical or technical incident; (d) a process for regularly testing, assessing and evaluating the effectiveness of technical and organisational measures for ensuring the security of the processing._

Recital 83:

 _In order to maintain security and to prevent processing in infringement of this Regulation, the controller or processor should evaluate the risks inherent in the processing and implement measures to mitigate those risks, such as encryption. Those measures should ensure an appropriate level of security, including confidentiality, taking into account the state of the art and the costs of implementation in relation to the risks and the nature of the personal data to be protected. In assessing data security risk, consideration should be given to the risks that are presented by personal data processing, such as accidental or unlawful destruction, loss, alteration, unauthorised disclosure of, or access to, personal data transmitted, stored or otherwise processed which may in particular lead to physical, material or non-material damage._

GDPR also mitigates the impact of leaks. Art. 5 requires that data is stored for no longer than necessary for the purposes for which it was collected. Art. 33 requires that the supervisory authority must be notified of any data breach within 72 hours. Art. 34 requires that data subjects be notified of any breach without undue delay. All of this is enforceable with heavy fines.

[https://gdpr-info.eu/art-32-gdpr/](https://gdpr-info.eu/art-32-gdpr/)

[https://gdpr-info.eu/recitals/no-83/](https://gdpr-info.eu/recitals/no-83/)

<comment><USER20> I can't see how this protects people. You can evaluate risks all the time you want, but unless you have exceptional security team, you won't get your situation improved beyond what's already been established in the industry. Unless you think about companies leaving their databases facing the public without password - but then I still can't see how GDPR would help there.

Requirements for post mortem actions are quite sensible though, but given arbitrary rules that is likely only going to be a cash cow for the governments, as even a second of the delay is undue.

<comment><USER22> The criteria for when such a delay is undue in Germany ("ohne schuldhafte Verzoegerung"), is if it would have been within your power and not incurring gross risks/costs (unless it's your fault for creating a situation where there are gross risks/costs) to have done the required action at an earlier time. It is not undue delay if you needed to sleep, or if your ISP just cut you off and you need to go into the city and get some other ISP to get you a connection, but it is your fault if you then sit around for a month, waiting for the ISP to get it ready, without the ISP getting the connection up soon enough. They would require you to go to the city and get a permit to string fiber across from the next hub to your building, if there was no other way to get it done sooner, due to e.g. there not being anyone with free time to dig up the street and fix the cable, or whatever.</comment></comment></comment></comment></comment></comment>


Logistical issues constrain UK's increased virus testing capacity - <AUTHOR> https://www.ft.com/content/a9d74c93-48a2-44a6-bf73-a4103c9101dd <comment><AUTHOR> > Donna Kinnair, chief executive of the Royal College of Nursing, said on > Friday that some ill and symptomatic nurses were driving for hours to > testing stations, only to be turned away because they did not have > appointments.</comment>


Dr. Erickson Covid-19 Briefing - <AUTHOR> https://www.youtube.com/watch?v=xfLVxx_lBLU <comment><USER2> "When you hear someone say the words "Let's just take a basic calculation and extrapolate it out", you're about to be standing on shaky ground.

Especially when what they're doing isn't actually extrapolation, it's statistical inference using a sample that's not random.

Context: I just watched a video where an urgent care doctor took the % of people testing positive, multiplied it by the total state population, inferred the total number infected across the state, divided the number of deaths by this, & then told people their odds of dying.

You can't say that 12% of the entire population has contracted COVID-19 just because 12% of the people tested are positive if the vast majority of those being tested have symptoms. Healthy people are underrepresented in that sample.

That would be like trying to estimate the percent of the entire population that's vegetarian by only surveying people at an In-n-Out burger joint. I mean the fries are great and everything, but…"

[https://twitter.com/DataRemixed/status/1254778325416828928](https://twitter.com/DataRemixed/status/1254778325416828928)

<comment><USER3> The calculation or extrapolation estimates the odds of dying if you contract the flu and exhibit symptoms. Why would you test subjects with no symptoms. that would be like taking chest x-rays for the entire California population to see the rate of ????.</comment></comment>