Surviving Django, if you care about databases - <AUTHOR> https://www.varrazzo.com/blog/2020/07/25/surviving-django/ <comment><USER2> I can't say I can recommend following almost any of this as "general" advice. It is, at best, advice from a complete expert _for_ complete experts who are already pushing the edge of what django should be used for, and who never expect to have to hand off their project to an average web developer. I've had to come to terms with the fact that the average web developer _is_ going to screw up e.g. writing their schema by hand in sql. And I know I'll likely be the person inheriting this two generations of developer after the original wizard left.

For an average web developer, someone just getting into django, or someone who isn't yet running into the limitations that the author apparently is, this is a recipe to just make a huge mess.

It's also very _specific_ advice. Recommending ignoring cross-database concerns.. I'm sure that's fine if you're an end-user developer building and running your own product, but if you're considering writing a library or reusable app you'd like to be useful to other django users, it's probably worth a second thought.

It's ironic though that the author makes such a prominent mention of "You Ain't Gonna Need It", because that's exactly what I'd say about most of the pimped-up tweaks mentioned here. 99% of people aren't going to run into these limitations and will produce a much more maintainable app by following the standard django doctrine.

<comment><USER3> For my company I wrote internal django apps that are completely database agnostic. They can be run in testing with sqlite, postgres, oracle, etc.

<comment><USER4> That severely limits you though.

And I'm not talking about postgis, hstore or such exotic features. I'm talking about indexes, transactions, locking and other basic feature that is often missing or implemented very different across databases.

You may not need, say, GIST indices that help when sorting "tweets by recent activity" for your proof of concept or even your test-env but you certainly do for production, or else you are truly missing out. And such basic tools are avaialble in most database-engines, but differ very much in how exactly they are implemented, to be used and to be tuned.

<comment><USER3> Django supports indexes, transactions, and locking, and if I need to specify raw SQL I can do that during a migration say.

90% of what I need is already there. And if I need something special, I can add it in later.

<comment><USER5> It's not really about what Django supports, but about what all the databases you're using support. If you run it in testing with SQLite, you are not going to have the same set of features for indexing, transactions, and so on as you do in production with, say, PostgreSQL.

The parent comment mentioned GIST indexes; I'd add partial indexes and triggers as two features that can make a huge difference when used even in a very few places in an app (which is what we do in Zulip, on PostgreSQL.)

<comment><USER3> Sure, but does it matter when unit testing code or developing the 95% where it doesn't?

It matters for production sure if you start getting slowdowns and you can have migrations that do that. But for cranking out APIs on DRF, say, it matters a lot less than you think it does.

<comment><USER5> For that I guess it comes down to whether the features you're relying on are purely optimizations (like partial indexes, fancy GIST index types, etc.), or go to the semantics of how the code behaves (like triggers).

I think for us it's the case that all the fancy DBMS features we rely on in the core functionality of the app are pure optimizations. So if we wanted to run tests on SQLite, that'd work fine except for when testing the parts of the app that happen to rely on those fancy DBMS features that aren't only optimizations.

But I'd consider that a pretty unstable situation -- it'd mean that we had one test setup for most of the app, and then still needed another test setup (with Postgres) in order to test some parts of the app. When I've heard from people working on apps that use different DBMSes for test and production, generally they don't take this strategy and instead they just limit themselves to the lowest-common-denominator features that exist in both. You can totally do that (many people have!); but as berkes's original comment above said, if you do you're missing out on some really valuable features.

And if you ever want to use even a little bit of those fancy DBMS features, beyond pure-optimization index features, in some core part of the app -- boom, you can't test on the more limited DBMS at all.

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> In my 8 years of working on large Python codebases, I think the Django ORM is one of the best pieces of software I've had the joy of using. I often recommend it independent of the rest of Django (e.g. to people building Flask apps that need an ORM+migrations).

Especially when working on a team with many junior developers who don't know SQL (myself included in the early days), the Django ORM has been a shining beacon of reliability and consistency allowing us to do complex database schema changes without ever having to worry about messing them up or not being able to revert our changes.

I don't think I can say the same for any other ORM I've worked with, in particular I have really bad memories of trying to match Django's migration reliability and flexibility using SQLAlchemy + Alembic.

Also in regards to switching databases, the ability to swap out PostgreSQL for SQLite on smaller projects without having anything break is a feature I've relied on several times, and I'm extremely grateful maintaining that clean break in the layers of abstraction was prioritized by the Django team.

Obviously the author of this article has had different experiences (and has considerable authority in this field), but I caution that their experiences are not universal among Python devs.

<comment><USER7> I came here to post essentially this exact comment. Give me Django with only routing, middleware, the admin panel, and the ORM and I'd be happy. The time I've saved because of these things so I'm shipping code and not writing SQL by hand is astounding. I've made the decision to use Python and Django over _other languages and frameworks_ solely because of the ORM.

I honestly can't think of a single other piece of software that has actively saved me as much time.

<comment><USER8> > Give me Django with only routing, middleware, the admin panel, and the ORM > and I'd be happy.

Very much this. I tend to lean towards Flask for building simple APIs. I think it's easier and faster to start a project and get something done.

But I always end up missing the admin features and the far superior ORM, and always regret midway that I did not use Django instead.

</comment></comment></comment><comment><USER3> A couple of real world data points.

Two Fortune 500 companies I worked at had projects in Django that used native SQL to do migrations. This led to errors in prod during deployment.

The order of the scripts had to run in a proscribed order. They had to be listed on the deployment instructions. So the migrations would run, and then fail during a prod deployment for various reasons. Most notably a dependent script hadn't run in prod yet. They had run in the Dev/QA environments and everything had run just fine. Or Dev/QA had been hand tweaked and that change hadn't made it to the script during prod deployment because they were testing it for a month to see performance before moving to prod, etc... etc...

It was all ugly.

In one of the two companies they finally decided to stop handwriting SQL and started trusting the Django migration system, and the problem with deployments went away.

 _AND_ they still were able to hand write sql to optimize the performance of the DB if needed.

<comment><USER9> >Two Fortune 500 companies I worked at had projects in Django that used native SQL to do migrations. This led to errors in prod during deployment.

> The order of the scripts had to run in a proscribed order. They had to be > listed on the deployment instructions. So the migrations would run, and then > fail during a prod deployment for various reasons. Most notably a dependent > script hadn't run in prod yet. They had run in the Dev/QA environments and > everything had run just fine. Or Dev/QA had been hand tweaked and that > change hadn't made it to the script during prod deployment because they were > testing it for a month to see performance before moving to prod, etc... > etc...

How do companies get this right? My organization has all these problems as well and I lose 10-15% of my time chasing misplaced SQL scripts or SQL scripts with improper tweaks.

<comment><USER10> There are tools that help this: Visual Studio Data Tools, RoundhousE, Flyway/Redgate - I highly recommend investing time in building them into your workflows.

<comment><USER11> Alembic is pretty great for this as well

</comment></comment></comment></comment><comment><USER12> > The motivation for writing this article comes from knowledge sharing with > the team I'm currently collaborating with, who are using Django the > canonical way. I am convinced they can better use the tools they have. Let's > see if they can be persuaded to drop Django migrations...

I've built and worked on enough large Django projects to know that if you don't do Django the canonical way, then I don't want any part of the project. If I joined a team and saw this weird hodge-podge of custom migration framework and shadowing ORM fields I would nope-out immediately. It sounds like this post author just doesn't want to use Django's ORM. And that's fine, you don't have to.

<comment><USER13> There's nothing worse than random teams reimplementing a migrations system because they think they know better.

The Django migrations system is far from perfect, but it's light years ahead of anything a small team can build.

<comment><USER14> I don't think these practices scale well. One person becomes the default "migration person" (with processes that are likely not as well documented as Django) and the team becomes reliant on tribal knowledge.

<comment><USER6> How is that different from one person becoming the default "migration person" on a team that doesn't use Django?

<comment><USER13> The difference is that Stack Overflow has a ton of answers to even fairly complicated questions and that reading the Django docs is an order of magnitude easier than delving into some random person's badly written code.

</comment></comment></comment></comment></comment><comment><USER15> Daniele is a great guy and I deal w/ him regularly as he's the author of psycopg2. However the mistake he makes here is in the realm of database abstraction tools having the sole rationale of "database agnostic code". That's really not the main rationale, nor is it really "hide the database so people don't have to learn SQL". The main one is "automation of work". A lot of the comments here refer to situations where hand-rolling everything always was just too error prone. There's a thing we do in computers when a job is very repetitive, tedious, error prone, and follows a complete pattern that you can define algorithmically, which is, _write a program to do it!_. that said I'm sort of glad Django took the heat on this one and not SQLAlchemy so I can get to bed without obsessing :)

<comment><USER16> Thank you for creating SQLAlchemy and being so active on its mailing list.

</comment></comment><comment><USER17> _> Django needs it because a web framework not tied to a single database vendor is more valuable than one tied to a specific one - and that's fair enough. But you don't: your web program, most likely than not, will not have to switch from one database to another_

This is missing the forest for the trees.

Sure, my one app won't need to migrate from MySQL to Postgres a year down the road, but as a developer, I'd much rather be allowed to abstract away each vendor's idiosyncrasies and be able to focus on my Python code regardless of whether I'm using sqlite, PostgreSQL, MariaDB, MySQL, Oracle or even MS-SQL

<comment><USER18> The idea that some library can abstract away database concerns to this extent is wishful thinking.

You inevitably need to understand the SQL that your ORM is generating and what the database is actually doing. Otherwise you're flying blind.

<comment><USER19> Realistically what ends up happening is that the DB gets abstracted away to a painfully feature-free lowest common denominator.

<comment><USER7> Django handles this quite nicely. You get your basic lowest common denominator subset by default. If you want something more powerful, you can explicitly import a db-specific feature.

And for some features, you'll get a NotImplemented exception. In the ten years I've been using Django, I've only seen these a handful of times, and only when using SQLite.

</comment></comment></comment></comment><comment><USER20> This article is trying too hard to dress the author's limited experience up as absolute truth. For example:

> Have you got PostgreSQL in production, but you want to test with SQLite > because it's easier to set up? If so, your tests are just a tick-box > exercise: you are not testing anything remotely plausible and resembling > your live system.

This is not completely wrong but it's just telling you that the author has never worked in an environment where it makes sense. Anyone who has knows that this approach can be used to catch a high fraction of errors locally, while not precluding a full slower CI build.

> Even if you added some form of manual auditing to each save() method, it > will not capture changes made outside Django. It wouldn't be very secure > either: Django uses a single user to access the database so if someone > manages to hijack that user, they would be able to change the data in the > database and alter the audit tables to hide their traces.

Similarly, while it is true that Django defaults using a single user for everything you can easily configure multiple connections (containing, say, an issue with your normal web code but not your management commands), you can grant access to only SELECT and INSERT on the audit log table but not UPDATE or DELETE, etc. There's nothing with the separate approach, of course, but it's not like it's some great discovery that frameworks are not intended to cover every possible use-case and sometimes you want to turn them off.

The extended "I hacked around because understanding the migrations module seemed like more work and then I was too far in to reconsider" section has the same problem. Being surprised that the migration system triggered an update when you renamed a field is, well, exactly what it's designed to do. That's exactly why there is an entire section in the documentation which tells you exactly what to do in that situation and how to avoid data loss — in that case, if the problem wasn't making foo.bar magic (which it almost always is) you'd create multiple migrations to add the new column, move (and usually convert) the data, and drop the old column. You can even run raw SQL if there's something like an efficiency hack which makes it worthwhile. This takes a couple of minutes and is reliable.

<comment><USER21> > This article is trying too hard to dress the author's limited experience up > as absolute truth.

Sure you're not doing the same thing? As Vivekseth writes below, the author wrote psycopg2.

\--- Allowing the DB user that the web app uses to have schema modification privileges is a massive security hole. If you're not hacked, someone will eventually drop the production database.

So, migrations in anything but SQL are a Bad Idea.

<comment><USER20> Yes, you'll notice that I'm not saying his ideas have no place but that they're not universally applicable. For example, the security "hole" is trivially avoidable by using a different settings file to do migrations on a separate server/container which doesn't get normal web traffic.

If you want SQL, you can also generate it from your migrations and send it over to someone else to run. This is not uncommon in enterprise IT.

<comment><USER22> To reiterate the point of the GP, you said:

> This article is trying too hard to dress the author's limited experience

Saying the author has "limited experience" is condescending and untruthful given his career and accomplishments so far.

<comment><USER20> Limited does not mean he hasn't accomplished anything, only that he's not speaking for the entire community and wording his post that way doesn't add anything to it.

He's obviously very proficient, which is going to shape your perspective of what's easy and how much control you want just like the scale of the projects you work on and the number and skill levels of your team.

There's nothing wrong with his opinion - my objection is the overly broad framing. It would have been just as good as "here are some things which worked for us" and letting the reader decide whether they are in the same situation.

</comment></comment></comment></comment></comment><comment><USER23> Not mentioned in the article, but the author also built psychopg2 ([https://www.varrazzo.com/software/](https://www.varrazzo.com/software/))

<comment><USER24> Thank you. Relevant.

<comment><USER25> For those who do not know Django uses said library for PostgresSQL support. Mind you, if you don't like Django's ORM, you don't have to use it.

<comment><USER26> Isn't it the standard Python library for PostgreSQL? A co-worker just installed it for some work having nothing to do with Django.

<comment><USER25> It's the most popular, if you mean standard by that metric then yes, if you mean standard as in built-in to Python, then no (I only mention this because SQLite _does_ come with Python OOTB).

</comment></comment></comment></comment></comment><comment><USER27> This sounds like a "golden hammer" argument[1], basically saying that SQL can do everything Django can do, and if you know SQL _really, really well_ you can do powerful stuff which would be clunky in Django. Of course that's true, because it's true of any two tools which have overlapping functionality. But for the vast majority of things you need to do on a daily basis, and for developers with >10x more experience with Python than SQL it just doesn't make sense to use raw SQL any more than absolutely necessary. As soon as you use raw SQL in Django most of the _maintainability_ advantages of an ORM disappear.

All that said, I totally agree about throwing out DB portability for anything but the simplest apps (or libraries, of course). Portability is extremely costly in terms of which features we have available to and how maintainable the resulting code is, so it should only be a goal when absolutely necessary.

[1] [https://en.wikipedia.org/wiki/Law_of_the_instrument](https://en.wikipedia.org/wiki/Law_of_the_instrument)

</comment><comment><USER28> > Much of the complexity of this system is designed to give you a specific > feature: migrations abstracted from your database. In my opinion, if there > is something less likely to happen than writing a complex app to run on > interchangeable databases, it's the need to repeat the same history of > migrations.

Unless you have a dev, staging, and prod setup. Then there's a good chance you'll encounter a wide range of variation in the current database schema compared to your models. I use the ability to roll back and forward migrations _all the time._ Django ORM and the migrations (and the admin site) are _the_ features that make it impossible for me to justify the use of other frameworks or languages in the vast majority of cases. The productivity boon is just too much.

With that said, I have a personal policy of not letting the ORM get in the way. If I find myself reaching out to the documentation for how to write a particular query (something like window functions, for example), that's a good indication that I may be better off writing the raw SQL.

</comment><comment><USER29> I support a very large database, containing product data for a major US retailer. If I understand the history of this database correctly (I am not directly on the team that manages it, but my team supports them in various ways) the database began life as an MS SQL DB. Many years ago it was migrated to MySQL. Now, they are in the process of migrating it to Postgres. So yes, migrating data to different DB's does happen, especially when you consider an applications life over decades.

<comment><USER30> Why was the underlying database changed?

The most obvious reason I can think of being licensing concerns with both MS SQL and MySQL.

<comment><USER29> The first migration from ms sql to mysql was licensing costs, and the desire to host on Linux. The more recent plan to move to postgres is motivated in part by features and in part by standardizing on postgres.

<comment> grey-area Features that you wouldn't be allowed to use if you try to remain database agnostic.

DB migrations happen, but not often enough to use a lowest-common-denominator db agnostic library and try to remain compatible with all systems.

<comment><USER29> Well, that is another area the article got partially wrong. There are many postgres specific features included in Django, and other available via third party packages. And for most things that the ORM doesn't specifically support, it gets out of the way and lests you do them anyway, by executing your own custom SQL, right from the ORM.

</comment></comment></comment></comment></comment><comment><USER31> > And with comments! Explaining why a certain index or constraint exist!

You can put comments in Python files...

> With constraints named meaningfully, available for manipulation, not > auth_group_permissions_group_id_b120cbf9_fk_auth_group_id.

You can name your migrations with:

        manage.py makemigrations --name "prettier_name"

> In my opinion, if there is something less likely to happen than writing a > complex app to run on interchangeable databases, it's the need to repeat the > same history of migrations.

Maybe he never worked in a team? Or wrote any integration tests at all? Or implemented rollback? Because those are pretty much basic things for every serious project.

> I have a patch_db.py script that I have used, with small variations, in > several projects.

So you are basically wrote your own schema migrations tool, but inconsistent across projects, and you maintain it yourself. Not a good idea IMO.

<comment><USER32> > You can put comments in Python files...

only if the comment pertains to a django feature, not to a schema feature that cannot be expressed in django (e.g. a partial index designed for a specific query).

> You can name your migrations with:

In the example there is a foreign key name, not a migration name. It is persisted in the database, it's not ephemeral like a migration name, for which choosing a meaningful name has only a temporary value.

Just two factual corrections; for the rest our experiences diverge, that's fine.

<comment><USER31> > for the rest our experiences diverge, that's fine.

[https://twitter.com/kissgyorgy/status/1291609448809627649](https://twitter.com/kissgyorgy/status/1291609448809627649)

</comment></comment></comment><comment><USER33> This article misses the point of the django orm and migration system completly.

They are meant to allow for the dev of an ecosystem.

This is why you have so many django plugins, and why they are so easy to integrate and play together.

This is why it's easy to have a dev envs, tests, signals and api generation with django.

It trades performances and flexibility for this. It's nothing more than a classic technical compromise.

You may need raw sql later on in your django project. But if you feel like starting with it, just go with flask. Django makes no sense.

<comment><USER25> The other funny thing is how easy it is to write migrations in Python via the RunPython capabilities of the Django ORM you can migrate data over. I've yet to need to write any SQL aside from stored procedures and Database Views.

You can even export all the contents of your DB to JSON or YAML and import it in Django, so you probably very well could go from SQLite to MySQL / Postgres if you really wanted to.

Edit:

The other actually funny thing is, Django doesn't force you to use its own ORM. You can write code that doesn't use its ORM. You can override anything and everything.

<comment><USER31> > You can write code that doesn't use its ORM.

You can, but what's the point using Django then? Everything is built around the ORM API, that's how admin is automatically generating pages, etc, etc. You lose the main advantage of Django if you don't use it.

<comment><USER25> ASP .NET Has the same issue, but you could always override the admin bits. If you _really_ want another library to manage Django's database usage, you can go to town, nobody has bothered coding alternatives because the ORM is sufficient. Web frameworks are not what you use if you're even concerned with micro managing how you access a database, web microframeworks on the other hand are designed for that level of detail. Something like CherryPy or Flask is more what you'd want in that case.

</comment></comment></comment></comment><comment><USER34> Migrations can, and do, run arbitrary SQL. At the end of the day the migrations framework is just a SQL dependency graph, it can be used without models if that's what you wish. By itself that is way better than ad-hoc scripts.

<comment><USER2> > At the end of the day the migrations framework is just a SQL dependency > graph.

It's not _quite_. During operation it actually mutates an in-memory shadow of your model structure. And that's really clever, but if you've got a large set of models it can be _really slow_.

<comment><USER32> It also breaks in interesting way: I discovered just today that constants defined on the Model subclass are not available when you use 'get_model()'. I suspect methods wouldn't be accessible either?

<comment><USER35> As far as I know, this is by design; your migrations are supposed to operate on a frozen state of what your models and code _were at a point in time_.

If you would rely on code outside of said migration, you would be breaching that frozen state and potentially end up with unintended side-effects (e.g. running a migration created 2 years ago that imports your code that changed today). This is why you might have to sometimes copy-paste logic to your python migrations, but you also guarantee that the migration always runs the same way.

</comment></comment></comment></comment><comment><USER36> I wish I could work on these magical projects that people with blogs work on, in which everything works like it's supposed to do. In my experience, though, it's not unusual to be forced by management to migrate to a new RDBMS a year into a project. The last time it happened to me, we were using Rails and had very little difficulty migrating from PostgreSQL to MySQL. We didn't need to alter application code or rewrite tests and only had to manage config changes and migrate the data. I imagine the transition wouldn't be much different if we had been using Django.

</comment><comment><USER37> Seems like the author should've just used a much less opinionated framework. The last thing I want to do is choose a batteries-included framework and then take on the responsibility of maintaining my own system to manage SQL migrations. Also unclear where the author has worked, but I've had to replay a bunch of migrations in order on many occasions.

</comment><comment><USER21> The implication of the article is that if you're using a relational database management system, _you need to know SQL_.

I agree. I wholeheartedly recommend Stephane Faroult's _SQL Success_ as a beginner book. Read that with attention, and you'll be ahead of 95% of people messing around with RDBs.

<comment><USER38> Also agree, sql it's like javascript, it's almost on all projects I worked last 20 years.. (Django it's just another tool, and implementation detail).

</comment></comment><comment><USER39> A lot of problems stated are characteristic of any ORM not just Django. People pick ORMs specifically for trading convenience and tooling with loss of control. Any escape hatches that the ORM provides is by definition hacky.

These days when I know it's a long term project, I prefer laying out the domain entities as plain classes and wrapping database operations as class methods with raw database statements.

This lets me use the underlying databases to its full potential and gives me enough control without leaking database code into application logic.

</comment><comment><USER40> > How many times have you worked on a project and, after 1-2 years of > development, you have changed to a different database? > > > I can tell you > how many times it happened to me, I counted them: exactly never.

I guess this is the crux.

You hardly know in advance this is going to happen.

I experienced this transition (mysql>pgsql), though not with Django.

The ORM was perfect. If I recall correctly, only the customized SQL queries (only a few present) had to br adapted, everything else just worked.

11/10 would chose such a framework again.

</comment><comment><USER41> I've migrated a production app from mysql to pgsql although not with django but a different orm and language. The app also had some custom sql. The reason was for proper json support as we have a 2tb table with unstructured data (1tb at the time) that was slowing mysql down severely (most used table by far). A later version of mysql with json support was not available on aws so wasn't an option. This was about two to three years in so the app was actually almost finished and in maintenance mode.

It worked. It took many months and copying the data and doing the migration was a pita. The orm helped in that I had to modify fewer queries by hand, but modifying the queries we did have was trivial also. It's one of only two times off the top of my head that the orm was a net benefit rather than a liability in the apps that use it. Would I use an orm in the future in case this comes up? Hell no. I'd pick a proper db first, aka pgsql (that decision was before my time). But even if I didn't, I'd still prefer to convert the sql one by one from mysql to pgsql rather than incur the horrific penalties and extremely slow run time and development time the orm imposed on us. Developing with the orm was many times slower and the runtime was ten times slower than the actual query time due to hydration. Rewriting a few thousand queries is nothing compared to losing months to the slow development speed and the slow runtime speed which required a completely separate rewrite of most of our functionality alongside the orm just so we could serve api requests in a manner somewhat reminiscent of not completely slow.

So yeah, even when switching from mysql to pgsql, the orm wasn't worth it. It never is or has been.

</comment><comment><USER42> We've got a core product running on Django, and one thing the author doesn't mention is testing migrations. The migration system is wonderful, except for the difficulty in testing migrations. There's no sane/official way to do this. And it's such an important thing that I don't get why the Django crew haven't tackled it.

<comment><USER43> I've used django-test-migrations before and was really happy with it. [https://pypi.org/project/django-test- migrations/](https://pypi.org/project/django-test-migrations/)

<comment><USER42> We've played around with that and it's on the roadmap. But it just adds another set of fixtures to herd.

</comment></comment></comment><comment><USER44> Pretty good read! But I don't like how the author tries to characterize generic ORM's like Django's as "Portability at all costs". I don't think the point is that you can switch DB's a few years down the road if you choose. He's right, nobody does that. Ever :) The advantage is that you can _start_ a new project with whatever DB you already have, or are comfortable with. So, if you want to begin a new Django project, you aren't locked into using MySql, or Postgres, or what-have-you. It is true you sacrifice being able to leverage the full gamut of features a particular engine might offer, but that is a compromise you make with any ORM right? Or, if the limitations are to much to bear, just don't use Django ORM, it's not required, it's just the path of least resistance.

</comment><comment><USER45> I agree Django's migration system has a few unfortunate quirks (most notably the help-text-change-migrations), but it's also worked great for Zulip across over 300 database migrations, and I wouldn't recommend anyone do their own migration framework on raw SQL files by hand.

The issues mentioned here in this article are real (E.g. we have a lint rule blocking importing other code inside migration files), but the author is also ignoring the benefits of Django's system, namely that it's a well-documented system that does most of the work of managing migrations for you, and does a good enough job that the vast majority of developers working on a large software project (like Zulip, with 96 people who've contributed changes to database migrations) will do it correctly without special training just from the Django documentation.

And the downsides are certainly minor compared to the obvious downsides with rolling one's own system with raw SQL (E.g. the fact that nothing validates that the model definitions match the database schema in the author's proposal!). I wouldn't recommend anyone take the author's advice for what to do instead, but this article is helpful reading for anyone curious about what issues one will encounter after a few years working with Django.

> How many times have you worked on a project and, after 1-2 years of > development, you have changed to a different database?

FWIW, Zulip was originally developed with MySQL, and then about a year into developed we switched to Postgres. It also had a phase where we ran SQLite in development environments. Building on Django made our life a lot easier doing this.

> Once the migrations have been applied to all your testing and production > systems, and all your developers' databases, just remove them.

This is only true if you don't care about being able to use tools like `git bisect` where you might want to switch a development environment to older versions and back again (I do this often as part of).

> If you do it in a loop, it results in a "ripple load", meaning you'll run 50 > queries, each one fetching one record, while you could have run a single > query fetching 50 records, or just adding an extra JOIN to another query.

This is my biggest complaint about Django's ORM -- lazy fetching of foreign keys is great in a management shell but rarely desirable in production, and it results in inexperienced developers doing database queries in loops all the time. We address this with a bit of middleware that formats log lines like this e.g. `41ms (db: 10ms/2q)` (showing how much time was spend doing database queries and how many queries were done) -- which lets one easily identify problematic cases of this both in development and in production.

But if Django had a setting that just disables the lazy fetching of foreign keys feature (making accessing an unfetched foreign key an error), I'd turn it on for Zulip.

See also [https://zulip.readthedocs.io/en/latest/subsystems/schema- mig...](https://zulip.readthedocs.io/en/latest/subsystems/schema- migrations.html).

<comment><USER17> _> This is my biggest complaint about Django's ORM -- lazy fetching of foreign keys is great in a management shell but rarely desirable in production, and it results in inexperienced developers doing database queries in loops all the time._

I may be misunderstanding you, but isn't this what select_related and prefetch_related exist for?

<comment><USER20> Those are how you optimize it, but first you have to recognize the need. The easiest way to make this visible is to increase logging or install Django- debug-toolbar, so people can see when the query count jumps, but the best way to prevent regressions is to use something like the testing framework's assertNumQueries() method to prevent regressions:

[https://docs.djangoproject.com/en/3.0/topics/testing/tools/#...](https://docs.djangoproject.com/en/3.0/topics/testing/tools/#django.test.TransactionTestCase.assertNumQueries)

I've used that with a fuzzy number class which will say it's __eq__ to any number below a specified max so you can say e.g. my homepage should take less than 10 queries and not have to touch it for minor changes while catching when someone implements a new feature in a way which breaks your select_related usage.

</comment></comment></comment><comment><USER46> Interesting comparing this to Ruby on Rail's ORM, which takes a relatively hands-off approach. ActiveRecord will produce a migration for you when you generate a model from the command line, but otherwise migrations are entirely the developer's responsibility. And the way he suggests handling patches sounds a lot like how ActiveRecord does it: keep track of what migrations have already been applied and only apply newer ones, and for a new database just create the up-to-date schema rather than applying the entire history of migrations.

<comment><USER43> There's nothing stopping you from hand-writing Django migrations if you choose. Django compares the state of the models to the state of the database after all migrations have been run to determine if new migrations are necessary, so as long as you've covered all of your model changes in your hand-rolled migration you'll be fine.

<comment><USER46> Ah, good to know. I have no experience with Django and the article gave me the impression that Django migrations can't be handled manually.

</comment></comment></comment><comment><USER47> I made a custom Django system for my personal website in 2013. It became to old to work with my web host. So, now I just run the django locally long enough to run a wget mirror. Voilà, static content.

</comment><comment><USER48> This is tricky, on the one hand keeping everything related to the database close to or in the database is probably cleaner (in the clean architecture sort of way), as it eases a separation between the business rules and the data saving and loading.

On the other hand, doing this loses a lot of the sugar and development benefits of using the ORM (either Django or SQLAlchemy, which I prefer), that are often enough—and perhaps preferred—for 95% of projects.

If I were to choose, I'd pick the ORM, though trying to place it as high in the stack as possible.

</comment><comment><USER49> Terrible article, especially the part about migrations. The automatic migration generation of Django of course cannot infer exactly how you want your schema to change if you start renaming or altering columns.

The author's solution, instead of adding on to migrations then opts for completely abandoning a tried and tested migration framework for arbitrarily executing sql files? This is madness

</comment><comment><USER38> May Django work with multiple tenants on different postgresql schemas? Can Django migrations handle this in an easy way (better than sql).. When projects grow old (+10years), things trend to complicate (lot of hands, lot of changed requirements.. ), and that's when the advices on the article had a lot of sense.. on the end data is there and the model is the database..

</comment><comment><USER50> History Lesson (if you care about migrations):

TLDR; I disagree: Django migrations are the best, and make reusable apps possible!

I've been using Django since before they introduced migrations and it was hell! Django only had "sync_db" then which would create your tables initially, but then you were on your own to write and run SQL manually for any model changes. This became especially annoying when using 3rd party apps, because if those apps changed the schema, you'd have to manually update your database for the 3rd party changes.

Then came along django-south which was a reusable app which provided migrations and solved all of the above problems. It became almost the standard approach, and third party apps often provided south migrations.

South became so popular, and was such a big improvement to the development workflow, that it's codebase was merged into Django (after making some significant API improvements) to become Django migrations. I can still remember having to covert all the files in the migrations directory from South to Django migrations.

Now we have a defacto standard of migrations work, and they allow easy use of customer SQL. Seriously, if you hate python based migrations as much as the author does, write custom SQL but put it in a Django migration so you get all the benefits of being able to rollback migrations etc.

Summary: when writing all of your own code for a single project then there is only minor benefits from migrations. But they are still worth it! In fact, I write lots raw SQL in my migrations, eg for constraints, triggers, notify, and just modifying my data. If I run SQL, it goes in a migration.

But migrations shine the brightest for reusable first and third party apps, which lets admit, we all use in most projects. This allows a single reusable app to be used for all projects regardless of database. Also, as a consumer of a reusable app, I can let migrations do their thing, and not worry if an update will break my database.

</comment><comment><USER38> I would also add to the article, that when you try to alter a busy table add it a timeout (otherwise you will lock the world). SET statement_timeout = 50; Alter .... ;

</comment><comment><USER51> My advice is to avoid Django ORM altogether

</comment><comment><USER52> Probably they should try django-north instead of their patch_db.py

</comment><comment><USER38> Another thing important that the author doesn't mention is that if you try to keep your sql model on the postgres side.. you will be able to introduce other languages (for example to optimize hot paths)

I agree 100% with the author, if you check other open source projects (on other languages) they mostly are based on the patch version sql feature. Sorry, the migrations feature from my point of view it's absolutely over engineered..

I also agree with him, that doesn't make too much sense to keep Django+drf, when if you use fastapi+pydantic+asyncpg you got all the real innovation on the python side..

Also it's not a good argument that following the Django religion will let to proper future maintenance.. think on channels and the big mess around them :)

Sorry, I'm perhaps a bit biased, started with Django 0.9 till 1.9.. and right now I'm not using it for anything new.. on the way I swapped for go, and later come back to python thanks to asyncio.

<comment><USER29> What mess around channels are you talking about? I've never heard anything about such a mess?

Not everyone uses Django to build a REST API that connects with a javascript SPA. And even if I was, I would still be skeptical that FastAPI could do everything Django does for me. (I do like Pydantic a lot, and actually use it some times in my Django apps.)

<comment><USER38> Look at what will happen to channels on the next releases when Django will be async...

To have a websocked popping info from a redis pubsub doesn't make any sense all of this crazy abstractions.. it's just an async view reading from aioredis..

If you need an orm (a real one) use sqlalchemy.. the Django orm has it's own design flaws that a lot of of dbas will complain about it..

Anyway, latest thoughts I have it's that it's better to keep you model logic as near as sql as you can.. you will swap before python or Django than postgresql :)</comment></comment></comment>


Comprehensive and Practical Inferential Statistics Guide for Data Science - <AUTHOR> https://www.analyticsvidhya.com/blog/2017/01/comprehensive-practical-guide-inferential-statistics-data-science/ <comment><USER2> If you really want to get into statistics I recommend this channel: [https://www.youtube.com/channel/UCFrjdcImgcQVyFbK04MBEhA](https://www.youtube.com/channel/UCFrjdcImgcQVyFbK04MBEhA)

Since I'm not great teaching, I've used this videos at work to make people understand stuff we've been doing (mainly programmers) and it's been great. This guy explains everything without making many assumptions and step by step.</comment>


Sequoia To Color Labs: Not Since Google Have We Seen This - <AUTHOR> http://blogs.wsj.com/venturecapital/2011/03/24/sequoia-to-color-labs-not-since-google-have-we-seen-this/ <comment><USER2> "Not since Google have we seen this"

I'm sorry, but this is the biggest line of bullshit I've seen in a while. Here's the thing: no one saw Google early on. Google happened, more or less, organically. They implemented a better algorithm and gained a lot of users by providing better results.

Color appears to be Instagram + location awareness. Awesome. I can't wait to use it, but to call it "Google" is just ridiculous.

I'm skeptical that this product is even going to be a success -- if I'm 150' from someone, why not just walk over to them, snap a group photo, and post it to my Twitter/Facebook/Whatever -- and even more skeptical that someone like Facebook won't clone the idea, roll it out to their 500 million users. Instagram already supports location detection. How long until "near me" shows up as a feed option?

Here's the real rub. Currently, there are 143 one & two-star reviews in the App Store. There are 49 four & five-star reviews. That's not good. Anyone claiming this is the next Google is either straight up lying to generate interest (It's the Next Google!™) or has no idea what they're talking about.

<comment><USER3> Give the Sequoia guys the benefit of doubt. They probably have seen something more (related to the product) that the rest of us havent or might be privy to other features in the pipeline. Obviously, they're a bunch of smart people who have backed some of the biggest companies in the tech landscape.

In short, dont be so hasty to judge.

<comment><USER4> I've been hearing that about Twitter for years now, and I'm still waiting.

<comment><USER3> While I am no longer active on Twitter, I cant deny the fact that Twitter has become a massive distribution channel for content,links etc. There are two other such online distribution channels -- Google and Facebook. And they make lot,lot of money.

Twitter might still not have a viable business model, but they can experiment (now that they have a huge user base and data) and try to find one. While its not easy and obvious at times to find a viable business model, just reaching the sheer distribution scale as Twitter's is a huge hurdle in itself that very few incumbents will ever reach.

<comment><USER4> I'm not arguing that - Twitter has grown much more than I expected. I was referring to the "I'm sure they know things us mere mortals are not privy to" argument.

</comment></comment></comment></comment></comment><comment><USER5> I think the knee-jerk cynicism about valuations and raises is blinding a lot of people on HN. Sure, Color's 41M is a lot, and it is YAPSS (Yet another photo sharing service). But Reading the techcrunch article about it, it sounds like one of those "sum is more than the parts" things, like the iPad or Google search results.

The integration of all the sensors and data to automatically determine your connections and maintain it for you _with no work_ could be huge! Don't be fooled by the fact that it started with photos - that's a convenient vector to start with and fuel growth, but there's nothing stopping them from hooking into phone and sms messages to make the implicit social connections more robust.

I'm not bullish on Color, but I'd take Color with $41M over Path, Instagram, or any of a zillion other sites with their raise. They have a huge chicken- and-egg problem, but isn't that what the money is good for?

<comment><USER6> _but isn't that what the money is good for?_

This just made me think that Color's best use of that money is using 10M to pay 1M people 10 bucks to use their app for a week.

<comment><USER5> That's basically what PayPal did.

</comment></comment></comment><comment><USER7> "Will $25 million help you get five years into one?"

Can 9 women produce a baby in one month?

<comment><USER8> Because, like pregnancy, businesses can't scale with more people?

I'm as skeptical as anyone about this insane valuation, but I don't see your analogy applying here.

<comment><USER7> It's a simplification, like any analogy. It's also a version of Brooks' Law[1], in case that wasn't apparent to all.

Businesses can certainly scale with more people. But this isn't a business, and to loosely paraphrase Andy Bechtolsheim at startup school this year, throwing money at problems is lazy, you should be throwing minds at problems. $41M in the bank will almost certainly be harmful at this stage of Color's development.

Also, even $41M is not going to (noticeably) speed up the surrounding ecosystem.

[1]<http://en.wikipedia.org/wiki/Brookss_law>

</comment></comment></comment><comment><USER9> Has anyone else tried using the android app? Its at 1.5 stars on the market, and on my device (G2), it did not work at all. It appears no QA was done on a number of device types.

I'm really surprised that Color is using Sequoia's brand for PR on WSJ, Techcrunch without ensuring that what users see on Android is not just plain awful.

I was / am really excited for this product, but this seems like a botched launch to me. Thoughts? Would it have been better for Color to delay launch? Will this have a negative impact on Color's brand? Sequoia's brand? Or will Color simply fix the Android app after a few iterations and provide a better and better user experience in true startup fashion?

<comment><USER7> I tried both the Android and iPhone version and couldn't take a single picture before it crashed. Tried multiple times on both.

<comment><USER10> Same here. It's crashed three times on my iPhone right after prompting me to take a picture of myself, with no option to skip.

</comment></comment></comment><comment><USER11>
      working to build out the infrastructure necessary to
      manage photos and videos **from what it hopes** will be
      hundreds of millions of users.

Emphasis mine. I especially like that part.

They've been tricked by their investors that they've already succeeded. That they already have won the market for photosharing apps, yet all they have is a domain name, a crummy unfinished product and a lot of PR. How can they possibly NOT get those millions of users with this kind of marketing hype and capital?

<comment><USER12> Early optimization and all that.

</comment></comment><comment><USER13> I don't get it.

I've installed the app and still don't get it.

I have no idea what problem does it solve?

I've seen <http://view.io> during SXSW - didn't work that well (not many people were using it). Concept was very similar - pictures of interesting things near you taken by other people with some descriptions. Nobody was declaring this app the next google.

Social network for mobile phones that is the next big thing???? I'm sorry but I don't buy this. Facebook will clone this using their location service + photo sharing functionality and there you go. 41mlnUSD down the drain.

</comment><comment><USER14> Am I the only one to think there is some kind of real-time Google Earth stuff going on behind the scenes, supposedly in their labs ?

$41M, I can clearly see how that would fund a face/shape/... recognition engine, reconstruct some 3D models from the images and then play with them.

Frankly the iPhone app just seems to me like a cheap way for them to get a bigger image dataset and begin telling users that all their photos will be public anyway.

</comment><comment><USER15> I don't get it. I just don't understand what is so big about this.

<comment><USER16> Hype. If they say it's big it must be so. $41mm funding announcement and the fanfare with it, they need a lot of early traction fast. And it seems this is how they are going about it.

<comment><USER15> But this is ludicrous even for a hype machine. "Not since Google" is absurd for a social photo and video app.

</comment></comment></comment><comment><USER17> Sequoia is not going to give $25 million away without good reason. It's just that folks like us can't see the good reason. Either the "good reason" is being hid from us for strategic reasons or Sequoia made a big mistake and the app/company is a dud.

<comment><USER18> Sorry, but this is really terrible logic. I have first-hand experience with watching VC's pour literally hundreds of millions of dollars into a company that I was certain was never going to make it (and which has since been sold off for pennies on the dollar). All along, people kept arguing that I must be missing something, because people wouldn't invest that kind of money unless success was assured.

<comment><USER17> I'm not saying that the "good reason" will result in a proven product. I'm just saying in their perspective they have good reason to spend that amount of money. And also it's Sequoia and they're not stupid.

<comment><USER19> There's a name for "good reasons" that don't result in proven product. They're called "bad reasons".

</comment></comment></comment></comment><comment><USER20> Am I missing something here or is there a typo? The first sentence says:

> Color Labs Inc. founder Bill Nguyen visited the Dow Jones office in New York > last week to talk about a big new idea backed by a $14 million round of > funding.

Later, the story says:

> With Sequoia's $25 million, $9 million from Bain Capital and $7 million in > venture debt from Silicon Valley Bank, Color has closed a $41 million round > of Series A funding before its product is even available in an app store.

Per TC yesterday & the WSJ in the same article, it's a $41M round.

[edit]

The author writes in the comments:

> Ty McMahan wrote: @Frank – The had secured only $14M as of last week. > Sequoia came in with an additional $25M and SVP added $2M more.

<comment><USER21> It was going to be 14, then it became 16 + 25 = 41.

</comment></comment><comment><USER22> There just has to be something else going on here. Right?

</comment><comment><USER23> Am I the only one who's first thought that came to mind when reading up on Color was the Sonar SuperComputer App in Batman: The Dark Knight? I don't mean this to be silly but the concept seems very similar in terms of data and perspective joined with location. If it reaches any type of criticality they'd have an absurd amount of visual data around location and time. That could've piqued Sequoia's interest.

<comment><USER24> Seems there are several of us thinking along these lines.

My version: "Color" + "PhotoSynth" = "OMG-level 3D world modeling"

Hype a critical mass of interest, throw a lot of CPU cycles at the data, port the results to Google Earth ... yow.

</comment></comment><comment><USER25> I think this is all a talent acquisition strategy.

</comment><comment><USER26> Investing 25 Million in something that isn't an issue or a need? Google brought an organised and simple search engine for the whole web... I don't get the Color app. We'll see how it evolves and grows because right now, it's just another funky mobile app.

</comment><comment><USER27> I thought SearchMe and Mahalo were suppose to be the next Google?

<comment><USER28> Cuil.

<comment><USER14> This is a serious contender in the shortest joke contest :)

<comment><USER28> Damnit, I should have left out the period.

</comment></comment></comment></comment><comment><USER29> I hope with 41M of funding and 18-24 months until they need to raise again that they are willing to throw out many revisions of their product until they get one that truly "revolutionizes" life. Right now so much is focused on hype and so little is focused on product/potential roadmap of where this could be in 5 years(compressed to 1 year).

</comment><comment><USER30> Am I the only person who thinks this could end badly simply because GPS is so spotty indoors?

<comment><USER31> Came here to say that. The iPhone seemed to be okay at using AGPS indoors but my Droid X fails miserably if I'm not outside. Good luck figuring out where I am, Color.

</comment></comment><comment><USER32> What nonsense, people have run out of ideas so VCs are desperately hoping someone else makes it. From their site: "Simultaneously use multiple iPhones and Androids to capture photos, videos, and conversations into a group album."

How revolutionary is this? Meh.

</comment><comment><USER33> After a thoughtful analysis I've come to the conclusion that the only reason they invested such amount of money is because of patents, not the shitty photo app. If Color Labs patented a way to show the strength of social relationships (with colors, or whatever) using mobile devices, then that alone might be worth billions in the future. At least for facebook.

</comment><comment><USER34> Reminds me of the pre-IPO dog and pony shows I experienced in 1999 when series 1 and 2 investors would tell mezzanine investors that "This is bigger than electricity, bigger than the railroads" with a straight face.

Sequoia is just talking their book to the greater fool. Nothing out of the ordinary here.</comment>


Map style changes on OpenStreetMap.org - <AUTHOR> https://blog.openstreetmap.org/2015/10/30/openstreetmap-org-map-changing/ <comment><USER2> I like the constant updates and community behind OpenStreetMap. I'm thinking, would there be a particular reason why a developer would choose Google Maps instead of OpenStreetMap for a new application? I don't know if there are any major flaws

<comment><USER3> The main reasons that someone would still choose Google, as I (a long-term OSM developer) see it:

\- Address search. OSM has lousy address coverage.

\- One single provider, works out of the box. OSM is much more flexible in the long run, but you can't just plug in an "OpenStreetMap API" and expect it to do everything for you. (That said, Mapbox is going a long way towards providing this.)

\- Satellite imagery. (Again, you can get this from a reseller like Mapbox.)

\- StreetView.

\- Public transport directions.

\- Familiarity.

<comment><USER4> How could OSM's address coverage be improved?

<comment><USER5> Go to [http://www.openstreetmap.org/](http://www.openstreetmap.org/), find your street, click 'Edit', login or create an account, and make sure your local area is correctly classified. You can mark an area as a building and give it tags to say what the address and street number is.

<comment><USER2> I don't think Google mapped every street single address to a set of lat/long?? There must be mathematical formula to map a street address, knowing how the city allocated street numbers

<comment><USER6> Oh dear. Maybe in some places, which has a sensible and consistant addressing format. But there are plenty of places where there is no rhyme or reason to it.

Have a look at Germany in OSM. That's what crowdsourced maps can do.

</comment></comment></comment></comment></comment></comment><comment><USER7> I don't understand this part:

> In the longer term there is the option to repopulate the database used by > the Standard map style so that it has access to all OpenStreetMap data tags, > not just a limited few.

How does a map 'style' repopulate the database? I get that not all tags can be displayed (e.g. many businesses have a website, but you don't display that unless you want to view the business's details) so why would a map need "access" to all data tags?

I have to be reading it wrong, but I don't see how.

<comment><USER3> There's the core OSM database (plain ol' Postgres), which contains all current OSM data plus all historical edit data. This is what you change if you click 'Edit' on OSM.

There's also the rendering database (PostGIS), which contains a subset of the current data, with freeform tags moved into Postgres columns, and with PostGIS geometries created from the raw OSM geodata. This is continually updated from the diffs of the core database. It's used by Mapnik (rendering code) to create the images you see on osm.org.

The proposed change is to add an hstore column to the rendering database, which will be populated with all the tags for each object from the core OSM database. This will require a complete reimport of the data.

<comment><USER8> Where would be a good place to start if I wanted to learn about adding additional data layers (in this case, US parcel ownership data) to OSM?

<comment><USER9> uMap is great for that:

[https://wiki.openstreetmap.org/wiki/UMap](https://wiki.openstreetmap.org/wiki/UMap)

If you want to do more complicated things, try leaflet, [http://leafletjs.com/](http://leafletjs.com/)</comment></comment></comment></comment>


ACLU Comment on Congressional Move to Allow ISPs to Sell Consumer Data - <AUTHOR> https://www.aclu.org/news/aclu-comment-congressional-move-allow-internet-providers-sell-consumer-data-without-permission <comment><USER2> Where was the ACLU when this policy was in place for 96% of the Obama Administration? The FCC just changed it a week before the last election.

</comment><comment><USER3> Some is really disconnected. It's either people who think privacy on the Internet or off matters or the politicians who think otherwise. Or maybe it's the lobbying that makes them turn a blind eye. If this somehow backfires and their personal surf history gets leaked, I wonder if they'll sit back and accept it as casualty of business or retaliate against the same ISPs.

</comment><comment><USER4> How exactly is my ISP going to sell my browsing data anyway? How would they collect it? Wouldn't something as simple as OpenDNS/etc thwart much of those efforts? How would they even determine what is useful data and sort it out from all of the junk?

How is this better or worse than all of the data that Google and Facebook have been collecting on users for years now?

<comment><USER3> Don't ISP's enjoy some privilege for being a carrier? Aren't they not one of they start doing all this? Also, most people won't bother with <insert simple solution here> so ISPs aren't too worried about it probably.</comment></comment>


American Airlines says computer glitch allowed all pilots to take Xmas vacation - <AUTHOR> https://twitter.com/SweeneyABC/status/935942616167960576 <comment><USER2> In regard to the 1.5x hourly rate that is being offered there is an interesting quote from the AA union on the NPR article[0]

"In a post to its website, the union warned its members that because "management unilaterally created their solution in violation of the contract, neither APA nor the contract can guarantee the promised payment of the premium being offered."

[0] [https://www.npr.org/sections/thetwo- way/2017/11/29/567286235...](https://www.npr.org/sections/thetwo- way/2017/11/29/567286235/oops-american-airlines-accidentally-let-too-many- pilots-take-off-the-holidays)

</comment><comment><USER3> That's unfortunate for American, but glad they found a way to alleviate the issue in a positive way.

Unrelated; "Unions upset about not being able to work" seems to contradict the anti-union sentiment that they just protect crappy employees</comment>


How to Send a File - <AUTHOR> https://blog.xkcd.com/2019/08/26/how-to-send-a-file/ <comment> _o-O-o_ Interesting. Never even knew XKCD had a blog until now. I love the way you can read all his posts on one massive page lol [https://blog.xkcd.com](https://blog.xkcd.com)</comment>


Microsoft announces $399 Xbox One without Kinect - <AUTHOR> http://www.theverge.com/2014/5/13/5713462/xbox-one-without-kinect-399

<comment><USER2> I should keep myself from buying all these stuff from day-one prices..</comment>


German armed forces warn that carbon fiber might be cancer-causing - <AUTHOR> http://www.ndr.de/nachrichten/niedersachsen/Bundeswehr-warnt-vor-Krebs-durch-Carbonfasern,cfk114.html <comment><USER2> Since the carcinogenic effect of asbestos is mostly caused by it's physical properties[1]; anything that resembles the shape of asbestos' micro-particles is also highly suspected to be carcinogenic.

 _The shape, size, and adsorbing nature of the fibers also appear to be critically important. Recently, doubts have arisen concerning the safety of commercially available carbon nanotubes,[2] which may possess the same carcinogenicity as asbestos fibers because of their similar characteristics. Ample care has to be taken to prevent a tragedy similar to the one caused by asbestos exposure._ [1]

[1] [http://www.med.nagoya-u.ac.jp/medlib/nagoya_j_med_sci/7112/p...](http://www.med.nagoya-u.ac.jp/medlib/nagoya_j_med_sci/7112/p001-010_Toyokuni.pdf) [PDF]

[2] [http://www.nature.com/nnano/journal/v3/n7/abs/nnano.2008.111...](http://www.nature.com/nnano/journal/v3/n7/abs/nnano.2008.111.html)

On making nanotubes less dangerous: _Shorter is better_ [http://onlinelibrary.wiley.com/doi/10.1002/anie.201207664/ab...](http://onlinelibrary.wiley.com/doi/10.1002/anie.201207664/abstract)

<comment><USER3> It's probably worth making the distinction that most people are exposed to "Carbon Fiber" as CRP[1], not "raw carbon fibers". And it is the ~aersolized bits raw carbon/dusts (the subject of this article) that are problematic.

_________

[1] The german title is "Bundeswehr warnt vor Krebs durch Carbonfasern", and From the German wikipedia page "Carbonfasern" ...

 _The fibers are mainly for the production of carbon fiber reinforced plastic used (CFRP = Carbon Fiber Reinforced Plastic). Derived from the English, the abbreviation CFRP (is english Carbon Fibre Reinforced Plastic) used._

<comment><USER4> Much like asbestos siding in that regard, where its not exactly being pumped into the air by most people, but people who work with it, for example sanding it, create a health hazard.

<comment><USER5> And indeed one way of making existing asbestos building materials safe (or at least, safer) is by spraying plastic on them to encapsulate them.

</comment></comment></comment></comment><comment><USER6> In another life I had to take aircraft crash response training, and was in charge of recovery (salvage, battle-field repair etc). The response to an aircraft with composite materials is totally different. With normal construction we could get to work as soon as the fire, was out and any hazardous liquids were taken care of. Crashes involving damage to composite structures required that the damaged areas be sprayed down with a varnish like chemical by the same troops who handle Chemical weapon response (they had special suits). Then there would be air particulate tests before we could move in. Here is a military study: [http://www.dtic.mil/dtic/tr/fulltext/u2/a420193.pdf](http://www.dtic.mil/dtic/tr/fulltext/u2/a420193.pdf)

<comment><USER7> A long time ago a coworker told me about this. He told me it was because the dust was conductive and would wreck havoc with electronics.

<comment><USER8> I've heard an anecdote along these lines, too. It was about a lathe that had been exposed to carbon-fiber dust starting to turn when supposedly off.

</comment></comment></comment><comment><USER9> Formula 1 and other open wheel racing drivers have been exposed to carbon fiber dust (from the carbon-carbon brakes) for a few decades now. There is some research going on with former racing drivers.

E.g. former F1 driver Mika Salo underwent surgery where his lungs were examined to assess the effects of repeated exposure to burned carbon fiber dust (this was several years ago). Unfortunately, I do not have any links to sources nor do I know the results of the research.

Another big question mark is the health and environmental effects of graphene. There is a lot of research going on in applications of graphene but only now there have been research projects into possible negative effects on the environment.

<comment><USER10> Here is something: [http://news.bbc.co.uk/sport2/hi/motorsport/formula_one/41211...](http://news.bbc.co.uk/sport2/hi/motorsport/formula_one/4121126.stm)

Apparently the surgery was unrelated, they found lots of carbon brake dust in his lungs purely by chance. No conclusion on whether it is dangerous or not.

Edit: grahamel was quicker :)

<comment><USER9> I recall him telling (in a Finnish commentary for a F1 race broadcast) that he went to another surgery to examine the carbon fiber dust in his lungs. But this was almost 10 years ago, I can't remember the specifics.

<comment><USER11> Can you wash the carbon fiber dust out of the lungs?

<comment><USER12> Washing air-breathing lungs with water sounds problematic, although with fluro-carbons and the prevalence of lung cancer I am not going to say no-one hasn't tried !

<comment><USER13> You could wash one lung at a time. Or use extracorporeal membrane oxygenation.

Water in an of itself won't hurt lungs as long as the patient gets enough oxygen.

<comment><USER14> "Water in an of itself won't hurt lungs"

Yes it does: [http://en.wikipedia.org/wiki/Drowning#Secondary_drowning](http://en.wikipedia.org/wiki/Drowning#Secondary_drowning)

If you ever get saved from near drowning, you are still not safe.

</comment></comment></comment></comment></comment></comment></comment><comment><AUTHOR> A research facility of the german armed forces warns that carbon fiber once burned at high temperatures is transformed to micro-sized particles having effects on the lungs comparable to asbestors.

</comment><comment><USER15> The USAF was concerned at one point that should an F-16 burn on a runway, it would release carbon-fiber strands which would float around the base and short out electrical equipment (communications gear, phone switch, power generation, etc).

Seems like the concern was correct, but misplaced. We had a plane catch fire on a taxiway and burn, and no electrical mayhem resulted. Perhaps we should have worn our gas masks -- but we didn't know at the time.

<comment><USER16> Sounds like that they were aware of 'Soft Bomb' technology and were worried that an accidental soft bomb could go off via the F-16's fiber/composite materials burning.

[http://en.wikipedia.org/wiki/BLU-114/B_%22Soft- Bomb%22](http://en.wikipedia.org/wiki/BLU-114/B_%22Soft-Bomb%22)

Interesting, that a plane from 1988 had warnings and concerns for a bomb that wouldn't be used until 1999. I wonder when the US first had one ready to go. What year was your experience?

<comment><USER15> 1983-85 at Hahn. We were the first base in USAFE to get the F-16 (A & B models).

Now it's a civilian airport where RyanAir tricks people into thinking they're flying into Frankfurt -- an additional 90-minute bus ride away (there's a reason why the airfare is so cheap!)

</comment></comment></comment><comment><USER17> While in undergrad I spent about 5 years working with carbon fiber in a motorsports setting. I did a lot of cutting and sanding without a mask. There were weeks I would blow my nose 3 days after working on a part and still find black specks. I guess I'll find out in a few years...

<comment><USER18> According to the article, this is the actually "good" variant: Normally, the dust is getting caught in the nose and does not reach the lungs. However, these fibres break down during a fire at temperatures of 650°C and become much smaller. This smaller size allows them to enter the lungs and potentially cause havoc there.

Put another way, every "black speck" you found did _not_ reach your lungs (although I doubt that this is particularly healthy, either…) :)

</comment></comment><comment><USER19> From the Google translated version of the article it looks like it's only dangerous when burned.

<comment><USER20> To be fair, most things are dangerous when burned.

<comment><USER13> > To be fair, most things are dangerous when burned.

That's not at all true. Some things are more, some things are less.

Incompletely burned plastic? More. Full burned? Less. Gasoline? Less. Mercury? More. Arsenic? Less.

</comment></comment></comment><comment><USER21> I was in the US military in the late 80s and early 90s and we were told then that carbon fiber used in aircraft may be cancer causing. Nothing new here.

</comment><comment><USER22> Translated highlights:

Carbon fiber once burned at high temperature breaks down sufficiently for it to penetrate mucous membranes. The risk highlighted by the Germans here relates to the ash left over after the fire. i.e. Hinting a potential long term dangers / contamination from the ash floating around.

</comment><comment><USER23> Note that this article (at least from the translated version) seems to be implying that the risk is around burning or particulate carbon fiber. I would definitely believe that.

The epoxy is also pretty nasty too. If you don't wear the right personal protective equipment when using it, you can quickly become sensitized to it. I don't know what that means medically, but it can't be good. I have always worn the right gear when building things, but it is clearly harmful to you without proper ventilation and separation from your skin.

I love composites, but they definitely have downsides.

<comment><USER24> I can confirm this. They say that the dust becomes dangerous above 650°C. The fibers become even smaller and reach the lungs.

</comment></comment><comment><USER25> I find it interesting that this article was yanked from Hacker News just as quickly as I saw it. I clicked on the link, wanted to go to the discussion and had to find it through search in spite of its high vote count and recent submission.

<comment><USER26> Hacker News is an English-language site. This is not to disparage content in other languages; it's just what HN is. Posts not in English mostly get demoted, unless they're of major significance and nothing is available in translation.

</comment></comment><comment><USER27> And next week our great Bundeswehr Research Facility will tell us that: smell of burned oil can make us ill and eating of weapon bullets can cause diarrhea.

</comment><comment><USER28> Though I'm all in to scientific development and the benefits of these new materials, I still want to point out, that nor carbon-fiber or graphene exist in nature in large quantities for more than one reason, while diamond and graphite do. Now the question should be .. why is that?

<comment><USER29> Likely because they require complex structures processes to create, not just pressure and time and heat. "Danger to others" is not a factor in nature's selection process.

<comment><USER30> Danger to others that share a gene line IS part of natural selection. That's why most creatures don't cannibalize within their family/pack for instance.

<comment><USER31> That's not what the OP was talking about however. The fact that carbon fibers are not present in nature only tells us that we didn't have to evolve anything protecting us from it if it's really dangerous (since it was never a selection criteria). That doesn't mean it's necessarily harmful though.

<comment><USER30> Disagree; OP was talking about why no animal in nature produces carbon fibre, given its dangerous to others nearby. Was it ecological pressure? I answered, yes that's possible.

<comment><USER31> Carbon fibers are pretty complex to produce, that alone can explain why they're not naturally present in nature. It would be similar to wondering why silicon microprocessors are not naturally synthesized by nature.

It doesn't necessarily mean that evolution ruled it out, just that it may never have considered it in the first place because it never came to exist under the stochastic process of evolution.</comment></comment></comment></comment></comment></comment>


VMware moves into mobile virtualisation - <AUTHOR> http://news.zdnet.co.uk/communications/0,1000000085,39547412,00.htm

<comment><USER2> Where's the benefit in this? Is mobile platform hardware evolving so fast that the only way to keep existing code working on new devices is to simulate the old?</comment>


Windows Server, version 2004 Now Available - <AUTHOR> https://techcommunity.microsoft.com/t5/containers/windows-server-version-2004-now-available/ba-p/1419194 <comment><AUTHOR> 'Here's how you can pull the new Windows Server, version 2004 base OS container images from MCR:

docker pull mcr.microsoft.com/windows/servercore:2004

docker pull mcr.microsoft.com/windows/nanoserver:2004

docker pull mcr.microsoft.com/windows:2004

[https://hub.docker.com/_/microsoft-windows-base-os- images](https://hub.docker.com/_/microsoft-windows-base-os-images)</comment>


Paradoxical Gear Sets - <AUTHOR> http://jacquesmaurel.com/gears

<comment><USER2> That's beautiful stuff. Gears are fascinating in their own right, these gears doubly so. As a kid I would disassemble (for the most part, sometimes I managed to put things back together again) all kinds of stuff and my favorites were alarm clocks and regular clocks. I must have destroyed what today would be worth a small fortune of clocks. Gears always fascinated me, they look so simple and yet they're quite complex.

The beginnings of the industrial revolution were grounded to some extent in the clock making industry, to aid the sea going nations. Cutting the gears for those clocks required a degree of precision that had not been seen for a long time and from that clock making industry we got to portable clocks (watches), mechanical computers and so on.

</comment><comment><USER3> They just love it in French engineering schools. They bring it on the day of the test, if you make an odd number of sign error you completely miss the trick, if you check your results for general coherency you'll never believe they are rotating in the same direction and you'll loose quite some time checking your stuff.

</comment><comment><USER4> See also: [http://www.shapeways.com/model/458290/magic- gears.html](http://www.shapeways.com/model/458290/magic-gears.html). 3 identical gears that, depending on the way they are assembled on their axes, behave like normal gears or paradoxically.

</comment><comment><USER5> Oh, I see. Normal gears can be flat, but paradoxical gears have to be 3D, they're like a pair of corkscrews driving each other. Right? Is there friction at the points of contact?

<comment><USER6> Paradoxical gears both rotate in the same direction.(e.g. both clock-wise)

Normally gears rotate in opposite directions, meaning the gear on the left rotates counterclockwise and the gear on the right rotates clockwise(or vice- versa). It just so happens that any paradoxical gear has to be "3D"(no 2D solution), there are "normal" gears that are 3D though.

</comment></comment><comment><USER7> Are they still gears if they are powered independently and the only connection between them is that they may come in contact without blocking each other's rotation?

<comment><USER8> I think that one is driving the others (in the first video it looks like the motor is only powering the middle gear)

<comment><USER9> The "Paradoxical Gears Explained" video shows a belt behind the gears. That suggests it's not really the gears driving each other.

<comment><USER10> They are driving each other. Although not well explained, the belt is just used to demonstrate how the angles are determined. This is a little clearer (although still hard to follow) if you watch the "Standard Gear Explanation" video first: [http://www.youtube.com/watch?feature=player_embedded&v=lWuKl...](http://www.youtube.com/watch?feature=player_embedded&v=lWuKl4rG2HA)

<comment><USER7> You're right. These 3D printed gears are clearly driving each other: [http://www.youtube.com/watch?v=qGAnmRb66s0](http://www.youtube.com/watch?v=qGAnmRb66s0)

<comment><USER11> Imagine screwing in a screw. Rotation makes lateral movement. Imagine pushing in a sharp-ended screw into something soft. Lateral movement makes rotation. What I believe is happening here is that one screw is being rotated, generating a lateral force on the other screw, rotating it.</comment></comment></comment></comment></comment></comment>


Bead Sort: An O(1) Sorting Algorithm. [PDF] - <AUTHOR> http://www.cs.auckland.ac.nz/~jaru003/research/publications/journals/beadsort.pdf <comment><USER2> This is cute but it's only "O(1)" if you have hardware that supports the parallel "drop" operation and you represent your data in base-1.</comment>


QIRA: Timeless Debugging by geohot [video] - <AUTHOR> https://www.youtube.com/watch?v=eGl6kpSajag <comment><USER2> I met geohot at a starcraft tournament at Facebook probably like four years ago. I turned around and saw his employee badge and exclaimed "You're geohot!"

He then proceeded to tell me about his killer idea for a timeless, replayable debugging tool that'd run circles around IDA. Glad to see it's starting to come together. That guy is super interesting.

</comment><comment><USER3> RR from Mozilla is interesting tool too: [http://rr-project.org/](http://rr- project.org/)

</comment><comment><USER4> Microsoft has IDNA, which is usually known to customers as time travel tracing and has a lot of similarities to qira. Unfortunately only Microsoft employees have access to the bits to analyze the resulting trace files you get - it'd be really nice of them to open it up.

<comment><USER5> I came here to mention this specifically. I'd love, love, love access to it, but they won't even let fairly close partners have access. :\

</comment></comment><comment><USER6> I actually like it. Anyone has an idea on what happens when you try to qira a network based application. For example, imagine you try to debug a client but you don't know how server behaves (TCP Stack etc...). It might be a possibility that not catching all states that client might have.

</comment><comment><USER7> "I was the first person to unlock the iPhone"

It's unfortunate that he actively spreads this lie.

<comment><USER8> @comex: "sounds like geohot all right. [https://t.co/busnrUz2hX"](https://t.co/busnrUz2hX")

@geekable: "@comex If I'm reading this correctly, he's bragging about stealing Vicarious IP. Great plan there"

@littlesteve: "@geekable @comex that's exactly what he did, again sounds exactly like him"

@SylerClayton: "@littlesteve @geekable @comex Yeah, no surprise there. I respect some of his past work, but would it hurt the guy to be a little less cocky?"

@littlesteve: "@SylerClayton @geekable @comex he has always been a little theify, it's his arrogance of "I don't need a team I can do it better"

@SylerClayton: "@littlesteve @geekable @comex Didn't he leak some of @Mathieulh's work dumping metldr on the PS3 or did he do all by himself?"

@Mathieulh: "@SylerClayton @littlesteve @geekable @comex he didn't do shit, I trusted him with my work and he leaked it. Then I had to document it all just to prove that it was indeed mine, even though I had not planned a public release that soon."

@marcan42: "He also gave no credit for our ECDSA fail research until I complained to him."

@littlesteve: "@marcan42 @SylerClayton @geekable @comex @Mathieulh that's our geohot right there"

@marcan42: "And AIUI he was the "first" to unlock (not JB) an iPhone but based on others' work."

Source: [https://twitter.com/comex/status/709868908161929216](https://twitter.com/comex/status/709868908161929216)

@comex, @Mathieulh and @marcan42 of @fail0verflow are very respected members of the community. I worked personally with @comex to develop the kernel exploit used in [https://github.com/wiiudev/libwiiu](https://github.com/wiiudev/libwiiu). I can attest that he's one of the nicest and smartest hackers I've ever met and helped me learn a great deal about exploit research and development. Take these words with a grain of salt if you must, but I wouldn't immediately disregard the statements of @geohots peers regarding his character...I'm not contesting that he's brilliant (he certainly is and I can respect that). I'm just saying he seems to be a bit of an egomaniac and sociopath.

</comment></comment><comment><USER9> Wow, the guy definitely is a genius, but boy does he seem annoying in that video.

<comment><USER10> Eh... I'll take the good with the bad. I have massive respect for the guy.

<comment><USER11> Funny I tend to think is way overrated and people are conflating works of many under his name (for instance he didn't crack the ps3).

ps: first time I see him speak longer, he's pretty energetic and inspiring I have to admit.

pps: the second part I see he's still in his battle-mindset again fakely valued tech companies.

<comment><USER12> True, but it's not like he is a total fraud, for the PS3 he was the one behind the memory glitching bug, which was an important step forward.

<comment><USER11> Fair enough. To be honest my issue comes from the self made man self driven car - david vs goliath thing he conveys often. I understand the fact that businesses have suboptimal structures and negative incentives in terms of engineering and technical prowess. And we all want real and beautiful innovation to happen as quick as possible. But his mashup of computer vision hooked into the electronic steering control was way less impressive to me than to the iphone/ps3 era crowd. Especially considering the social aspect of a ton of steel moving in public.

<comment><USER10> He gave a 15 minute speech on reverse engineering, then spoke for 4 minutes about self-driving cars...

In such a short time, what were you expecting him to achieve? (re: "social aspect of a ton of steel moving in public")

<comment><USER11> Not talking about this talk in particular, but the other interviews and his overall message.</comment></comment></comment></comment></comment></comment></comment>