Database versioning best practices - <AUTHOR> http://enterprisecraftsmanship.com/2015/08/10/database-versioning-best-practices/ <comment><USER2> I agree. Rails database migrations get so many of these things right: every change is a script, the database is versioned (via schema_migrations). I really like that the database is the sum of a list of deltas. I've worked with people who argued it'd be better if you could just write a declarative schema description by hand (like schema.rb), and have Rails automatically adjust your database to meet that description. But if you already have data you need to migrate from the old to new structures (e.g. changing a 1-to-many to a many- to-many), that isn't going to work.

I'm thrilled he mentioned that once you deploy a migration script, you can't ever change it. I wrote the same thing here [1]. But I see it happen all the time, and it guarantees that your schema gets out of sync with others'. If I've already run that script, Rails doesn't know it needs to run it again.

One thing he left out is that you should write migrations so they work not just on the current version of your code, but on future versions. I see this cause problems on projects again and again. For instance a feature branch with a migration from day 1 with `Permission.create!(name: "foo", value: "bar")`, and then a week later the developer decided to rename the Permission class to Setting. By the time the branch was merged into master, the migration was broken. My solution for this is that a database migration should never depend on the application code, but only use direct SQL.

[1] [http://illuminatedcomputing.com/posts/2013/03/rules-for- rail...](http://illuminatedcomputing.com/posts/2013/03/rules-for-rails- migrations/)

<comment><USER3> I've arrived at a similar conclusion, with a few exceptions:

Never changing past migration scripts should be a guideline, not necessarily a hard rule. With Rails, I generally prefer to use ActiveRecord in migration scripts (rather than raw SQL) for the purpose of keeping things more readable (again, guideline rather than hard rule). As you mentioned, sometimes changes in application code can break past migrations, hence the need to occasionally alter a past migration (e.g., placing parts of the migration in a conditional). With practice, this actually encourages one to write relatively "future-proof" migrations, anticipating possible future code changes and creating each migration with as few assumptions as possible.

At times, as in your example, a migration may alter data (not just structure), and sometimes old migrations alter data in an obsolete way that isn't really helpful anymore. Occasionally (especially if the migration in question is time-consuming), I'll revise an older migration, but I always keep it around (so the version still exists) and add comments to note what is being removed and why.

One thing that really helps is regularly running all migrations in sequence. My test suite would routinely run them all, and I never use a schema.rb to load a schema directly. Vagrant dev environments are always built using all past migrations. Doing that regularly helps ensure that the current database schema is always a result of repeatable deltas.

One more thing: I avoid migration rollbacks whenever possible. I've seen them cause more chaos than relief in already-stressful situations. In practice, this means (1) making sure the codebase before and after a migration can handle the database state before and after the migration, (2) writing migrations that allow you to easily restore the database to a consistent state if they fail midway through, and (3) whenever possible, fixing a migration and redeploying rather than attempting to roll back changes. A rollback is, after all, a migration itself, prone to its own set of potential bugs and failures.

<comment><USER4> One solution to the problem of ActiveRecord code in migrations getting out of date, that I usually like more than either the solution of only using raw SQL (though often it's the _right_ tool for the job) or of editing migrations later, is to define a version of the classes you want to use locally inside the migration class. This is usually as simple as, eg.:

        class DoSomethingWithPermission < ActiveRecord::Migration
          class Permission < ActiveRecord::Base; end

          # ...
        end

It protects you from all sorts of tomfoolery – changes to the class name, its relationships, its validations, etc. – and doesn't require any change in style.

I also love the idea of migrations being forward-only, and deployed on their own (ie. without accompanying code changes). If you can pull it off, those constraints give rise to some nice properties: schema changes must work with existing deployed code, code changes must work with existing deployed schema, deploys can be done with zero downtime, botched schema changes can be fixed without rolling back code changes, etc. It is harder to do than the traditional style, requiring schema changes to be made in phases, and often involving triggers and/or views to keep things consistent, so its cost/benefit isn't necessarily a clear win.

<comment><USER5> Thanks for this -- defining a local copy of the class is brilliant!

</comment></comment></comment></comment><comment><USER6> I mostly agree with this but with one (and a half) modifications and one addition.

Instead of storing a version as an integer, I strongly prefer _naming_ each migration and storing the applied migrations in a database table.

Rather than relying on a single integer, I can simply write code that applies all the migrations which haven't yet been applied. This makes it much easier for the database to be modified in multiple concurrent branches without any merge pain.

At work we follow this practice and name our migrations something like "1-add- foo-table". If another dev makes a branch with a migration named "1-add-bar- table" then there's no conflict.

We also store our migrations as a directory rather than a single script. This lets us split things up into multiple files if needed. We also allow for both SQL and Perl (our main language) migration scripts in the directory, which is handy.

Finally, all migrations must be idempotent. In theory no migration should ever be run against the same database twice, but making our migrations idempotent is a little extra insurance.

For the curious, I wrote some Perl modules that help manage a system of migrations like what I just described:

* [https://metacpan.org/release/Database-Migrator](https://metacpan.org/release/Database-Migrator)

* [https://metacpan.org/release/Database-Migrator-mysql](https://metacpan.org/release/Database-Migrator-mysql)

* [https://metacpan.org/release/Database-Migrator-Pg](https://metacpan.org/release/Database-Migrator-Pg)

The system is designed to be extensible so you can build on top of it, rather than a fully encapsulated tool.

There's also Sqitch ([http://sqitch.org/](http://sqitch.org/)). It's written in Perl but is entirely language-agnostic to the best of my knowledge.

<comment><USER7> > Finally, all migrations must be idempotent.

That must be tough. Are you doing it by simply checking a "I've already done this flag"?

<comment><USER8> You'd have a database table where its entire contents explain what's been done to the database schema and the data migrations.

And then you make inserting a row named '3-add_customers_table' as part of the transaction for the migration. If this row already exists, the insert will fail, aborting the entire transaction.

Or something like that...

</comment></comment></comment><comment><USER9> I had a project from about 6 years ago that used this method of DB versioning. The project was quite large and took 4 years to make-- it had a very large database schema and required a lot of default data.

One day recently I pulled that project off the shelf and decided to see if I could get it running for nostalgia's sake:

SGS Server starting...

Expected database version: 57.

Found database version: 0.

Applying DB change script: 1...success.

Applying DB change script: 2...success.

(...)

Database updated.

SGS Server running.

After seeing my old project set itself up automatically with no grief, I felt an enormous amount of pride. I only wish my day job would implement a similar system.

</comment><comment><USER10> This can be a major pain for larger software projects. At work we have to contend with:

* A database so large that even a minimally empty one cannot be created from scratch in less than 10-15 minutes. This creates problems for CI, integration testing, etc.

* The developers spent a good chunk of the late 90s/2000s writing Oracle PL/SQL code; hundreds of packages and thousands of stored procedures with oodles of business logic.

* We store reports, pdf attachments and other documents etc all in the DB as well.

* Since we put so much stuff on the database, small problems and schema fudges tend to creep in over the years, which makes every customer database a little bit different.

* Oracle licensing can be very unkind and the upper management mandate that we can't use the oracle XE version even in development/testing.

We ended up using a combination of Flyway for schema changes, hand-rolled scripts to apply stored procedures and packages, and we had to roll a database provisioning pool as-a-service for developers, and it's still a massively janky and fragile setup. We really need better tooling for this.

<comment><USER11> Last I looked, Oracle was free for development and test use. Is that no longer the case?

<comment><USER12> You see the problem there is that development and test are gateway environments. They lead frequently to other environments such as staging, and production, at which point you find yourself dependent on Oracle's latest stuff and it can be time consuming and expensive to get yourself clean.

Next time you find yourself Jonesing for the latest features and someone offers you dev & test for free, just say no.

</comment></comment></comment><comment><USER13> Liquibase is pretty good for managing database versions

[http://www.liquibase.org/](http://www.liquibase.org/)

<comment><USER14> Oh, good old "programming in XML".

<comment><USER15> "Supports XML, YAML, JSON and SQL formats" \- from the front page of [http://www.liquibase.org/](http://www.liquibase.org/). What more do you want ?

<comment><USER14> As if programming in YAML or JSON is any better. And programming in platform- specific SQL, for all non-trivial purposes, ties you to the specific flavor of DBMS.

What I want is a DSL that is a platform-agnostic DSL: terse, typechecked, possibly "compiled" to some IR, with basics of semantic analysis, with support for (or at least awareness of) the quirks of the specific underlying platform.

</comment></comment></comment></comment><comment><USER16> I think this post only gives half of the story. The thing is you need to manage not just your schema (and reference data) but also your DB code - i.e. stored proc etc.

For schema, the approach recommended by the OP seems sensible - check diff scripts into VC and consider them immutable (see also [http://www.depesz.com/2010/08/22/versioning/](http://www.depesz.com/2010/08/22/versioning/) for a lightweight Postgres approach).

For stored procs etc, I think you need an approach much more akin to normal code - here you want to be able to leverage your VCS just as you do with normal code - so for these you want to consider them mutable.

One of the best presentations I've seen on this topic is:

[http://www.slideshare.net/OleksiiKliukin/pgconf- us-2015-alte...](http://www.slideshare.net/OleksiiKliukin/pgconf- us-2015-alter-database-add-more-sanity)

<comment><USER17> Stored procedures are usually considered part of the schema. They can be diffed. I know the Visual Studio comparison tool is able to diff databases but also generates code to check affected stored procedures for errors after a table schema change.

I would still recommend changing stored procedures through UPDATE scripts and having this change scripts immutable. (Ideally not just immutable but also idempotent and with forward and backward scripts to undo change.)

<comment><USER16> Curious why you'd recommend that. It seems to me that it costs you something (convenience of standard VC practice on code) and gains you little.

I'd recommend checking out the linked presentation above - for things like stored procs you have the option of installing multiple versions simultaneously (e.g. under different names or 'Schemas') - obviously you can't do that for the main schema itself. This is why I think a hybrid approach makes more sense.

<comment><USER17> It's something I've seen working in a few places, but I'll take your recommendation and review the slides for a better approach. You're right that this approach does lose some of the benefits of version control but then typically the SP schema changes are very closely linked to table schema changes anyway.

</comment></comment></comment></comment><comment><USER14> Back in the day, I wrote my very own database migration toolkit [1] with its special DSL and cool things like automatic rollback script generation and (limited) type inference. It never gained any traction, though. But I still use it in every project with great success.

[1]: [https://code.google.com/p/octalforty- wizardby/](https://code.google.com/p/octalforty-wizardby/)

</comment><comment><USER18> _> Don't use complex notations like "x.y.z" for the version number, just use a single integer._

I prefer to use two integers: major & minor version, semver style. Minor version gets incremented if old code can still use new DB (e.g. when adding a column). Major gets incremented when the change is not backwards-compatible (e.g. when renaming a column).

When you have to rollback a release not having to downgrade the DB can be a major time saver.

<comment><USER19> Why not timestamps + description? This way, even if you create migrations on different branches, you don't have version number overlap. Rails does that if I remember correctly.

eg:

201509102039-add-this-to-table.sql

201510220840-add-that-to-table.sql

<comment><USER18> Because version numbers can be handled automatically, descriptions can't.

Say, you have app v.20150701 running in production. Today you deployed new version and it ran migration script against production DB. Two days later you discovered a critical issue in your new version and are forced to roll back to the old app. Now, your 20150710 app opens the database and notices that it has 20150811-oh-we-did-something-to-the-db.sql migration applied. Can it use the database or should it fail/downgrade the DB? How do you tell if all you have is a timestamp and description?

IMO this is more important than solving branch merges because it helps me when my production system is down and there's pressure to bring it back online ASAP. Merging branches can be done offline in the cozy comfort of my development environment.

PS. Of course, the above scenario might be more or less relevant depending on your development and deployment workflows.

<comment><USER19> Interesting point, though I've never thought about this situation because I've never worked with software versioning. If I need to rollback, I have to check which migrations are present in the SCM repository in a specific revision that I need to rollback to

</comment></comment></comment></comment><comment><USER20> For our Scala web applications this is pretty much a solved problem, since Play Framework keeps track of database versions in an 'evolutions' table, and runs evolutions in order to bring the database up-to-date. We've also modelled our test snapshots runner after the evolutions format - so each test fixture contains queries for both set up and teardown, on top of the empty vanilla database described the evolutions, of course.

The trouble we encounter is extending outside software to meet our needs, specifically Wordpress. WordPress is a better-than-average CMS but like many CMSes it is very developer-unfriendly. I need to be able to snapshot the database from a single WordPress installation and version it along with an installation of the several WordPress Plug-Ins I am developing for a specific client. This way I can keep a repository reflecting the current state of a single site (with a specific set of plug-ins activated, and specific configurations for each), an exact copy of which I want to eventually deploy in a production environment. I've seen plug-ins for this, but for at least in one specific case (VersionPress) it seems like it is only available through an overly expensive 'Early Access Program' \- and it's been in Early Access for many months now, making it seem Vaporware-ish. Anyone know of a better, automated solution, other than dumping the MySQL database every time I make a change?

<comment><USER21> I just really wish the Play Framework evolutions feature allowed you to give a logical name to your scripts. It would nice to be able to tell at a glance what the high level purpose of each script was.

<comment><USER22> +1. This bugged me about the excellent Schema Evolution Manager. Ended up porting it and extending it to take an optional name to add after the date.

</comment></comment></comment><comment><USER23> Sure, I'll throw my hat in the ring here with a shameless self-plug.

I help maintain a database versioning/migration tool that my team has been successfully using for years now: [https://github.com/nkiraly/DBSteward](https://github.com/nkiraly/DBSteward)

The idea is that instead of managing a schema + several migrations, you just store the current schema in XML, then generate a single migration script between any two versions of the database (or just build the whole thing from scratch). The ideal use-case during deployment is to checkout the existing deployed schema into a different directory, then diff against the current and apply the upgrade script.

I've always found most migration solutions to be wanting, and while this approach has its downsides (things like renames can be hairy, need to explicitly build in support for RDBMS features), I do like it a lot more than the standard sum-of-changes approach.

The tool is still very much a work in progress, written in terrible 10-year- old PHP, and desperately in need of a proper rewrite and modernization, but it is definitely stable, safe, and production ready.

</comment><comment><USER24> Flyway does exactly this and enjoys widespread adoption [http://flywaydb.org/](http://flywaydb.org/)

<comment><USER25> Another happy user of Flyway here which essentially forces all of the described best practices onto your DB. If they only supported transactionless migrations, we'd be totally happy campers :)

<comment><USER24> Coming very soon :-)

</comment></comment></comment><comment><USER26> We use [http://www.dbmaintain.org](http://www.dbmaintain.org) at the Java shop where I work. It enforces all of the best practices in the article.

Something else that we do is to have the schema/mandatory data in a separate git project. This has the added benefit of having a separate deployable artifact from the application itself.

</comment><comment><USER27> We're using Delphix (off Oracle) to do our "every developer should have a database". For anyone not familiar with Delphix, it provides individual DB instances that are "forked" from the source DB. I'm assuming it's some kind of Copy On Write.

So far it's working ok, although we've not automated, snapshot/update/refresh off the master yet. The master is planned to be periodically refreshed from ProdCopy.

Would be interested if anyone else is using Delphix and can comment or link to anything on it. I've been meaning to poke around the REST interface but haven't had time yet.

<comment><USER28> We're using Delphix too. Complements an approach like this, doesn't replace it. We use Liquibase to support the schema-as-code concept; it's enforced by no one outside Sr. DBAs being able to make such changes directly in anything other than Dev (lower-than-QA) environments.

Delphix is terrific for having production-like data in your test environments ("like," because you'd better be masking sensitive information from prod before it gets written in test.

The issue we've had with Delphix is performance. It absolutely pounds on the storage system, and tends to suck up all available bandwidth and CPU that's made available to it. If you resource it properly, it's amazing.

</comment></comment><comment><USER29> It's all nice and good but there is an additional problem the article doesn't mention: how do you handle "exceptional" (unplanned) cases in your database version control? Sometimes you discover things don't work as you expected, or that there is a last-minute schema change you want to promote quickly to the production db. I find it hard to encompass these situations by modeling changes via forward/backward migrations (database schema diffs, like rails does).

At my workplace we operate a large-ish web service in our own datacenters (airline reservations). We're very happy with the way we version code: we've one single branch where changes arrives all over the clock; every second day this branch is deployed in a "pre-production" platform. Every week we take one of those versions, fork a release branch out of it, and make all necessary adjustments, i.e. a bunch of cherrypicks (changes arrived late but needed quickly) and possibly backouts (we found a bug and have no time for a fix):

        a--b--c--d--e   <--- what's in pre-prod
               \
                b*--e'  <--- what I want in prod

        b* = backout, e' = cherripick

Now, I really really would like to do the same for our DDL (database schema changes). But backouts and cherrypicks just don't work if you're versioning migrations, because you have to deal with what's already in the prod db (conversely, when you deploy software you just trash the old binary alltogether and replace it with the new one):

backouts, like b* above: you would hand to your DBA the sequence of patches a,b,c,b* ,e' which makes no sense because you're asking to apply migration b and its reverse b* (you should say "just skip b", but version control doesn't work like that)

cherripicks, like e' above: good luck remembering, in two weeks from now, that you've already applied e' and don't need to apply e.

At the end of the day, we -do- version our changes with forward/backward migrations, but we always end up managing backups and cherrypicks by sticking notes on the wall (or on the team wiki), which smells bad. I admit I never tried any of the popular solutions for database version control (liquibase, flyway) and I don't know how they work. But I feel the ideal solution would be to version the whole schema (not schema diffs), and have tools that at any given moment can compare the schema I have in my VCS and what is actually on the platform, and generate the necessary "migration" on the fly.

EDIT: formatting

</comment><comment><USER30> Django's South does this. [http://south.readthedocs.org/en/latest/tutorial/part1.html](http://south.readthedocs.org/en/latest/tutorial/part1.html) And now, the next version of that is built into Django core. [https://docs.djangoproject.com/en/1.8/topics/migrations/](https://docs.djangoproject.com/en/1.8/topics/migrations/)

</comment><comment><USER31> What about versioning data? For most projects we have SQL tables consisting of preconfigured data, that is essential for the project to work. (predefined templates, initial configuration settings) How do you go about producing an SQL difference script for your data, getting it under version control and then applying it at the push of a button?

Currently we are using the Toad for SQL tool for producing and applying both schema and data diffs between our development environments, but it's a Windows only tool.

<comment><USER32> Two things that may be relevant that I've done before:

1\. The simplest. Setup scripts for doing full backups and restores to/from S3. This is helpful for many reasons, but should also help your case. Diffs are an optimisation if your data is large enough, but I think many cases are easily small enough to just push and pull as a whole. Until you're at a significant number of gigs of data, I'd recommend trying this first.

2\. Record your data in an append-only format (this doesn't require a different database, just a strict way of doing things). This means you can always get a full history of any bit of your data. I do this for some scraping work (recording lots of noisy/unreliable values, so not losing old versions is important). If your data is stored like this, pushing around diffs should be easy (grab everything with a created_at time after a certain point).

</comment></comment><comment><USER33> We're using this approach on a large team and db but are struggling with having multiple major version branches in play at the same time and backporting of functionality and bugfixes breaking things by changing the order in which changes get applied. At some point, the problem remains that you're migrating the data in a live system, and that means the order of changes matters.

</comment><comment><USER34> Archiveopteryx (aox.org) handles database versioning in a very clean way. You can actually upgrade and downgrade your application and everything required to change your schema and dataset is contained within Postgres.

[http://archiveopteryx.org/aox/upgradeschema](http://archiveopteryx.org/aox/upgradeschema)

</comment><comment><USER35> I would like to add that it should be easy to blow away your db and recreate the structure with enough sample data so that the application works in a dev/test environment without having to pull in a db dump from some other machine. Depending on dumps drives me crazy.

</comment><comment><USER36> This touches on some very common issues related to migrations. It's very interesting to see that after all this time, effectively handling database schema change management can still be a challenge. I wrote about my experience with this topic several years back. I think some of these observations might still be useful to folks just digging into the problem: [http://robertelwell.info/blog/future-proof-db- changelog/](http://robertelwell.info/blog/future-proof-db-changelog/)

</comment><comment><USER37> Wow, I came in there thinking what we will learn. Turns out we do all of those points in our platform!

[http://platform.qbix.com/guide/database](http://platform.qbix.com/guide/database)

[http://platform.qbix.com/guide/scripts](http://platform.qbix.com/guide/scripts)

We use the actual database schema as the primary source of truth for the schema, and we generate the code for the models from it. We have scripts that upgrade the database schema for each plugin and app.

</comment><comment><USER19> I have a similar approach, but with timestamps instead of an incremental int.

I'd use a tool for this, but then I would need to change how I deploy the apps given that usually, it's best to run the migrations and then update the codebase to prevent exceptions.

In case there are new columns, the best solution would be: execute the migrations (if there are new columns, make them NULL), deploy the code, fill the new columns via SQL or code, if the new columns must be not-NULL, change the table definition and make the columns not-NULL.

Given the complexity I find better to do it by hand.

<comment><USER4> Yeah, timestamps seem to work much better, especially with distributed version control. Otherwise you have to coordinate who gets the next number in the sequence. There's still a race condition between two branches changing the schema in incompatible ways, but with a clean merge. But in my experience, code review, tests, and just developers running migrations day-to-day mitigates that issue.

</comment></comment><comment><USER38> I am sort of new to this stuff but I believe Django essentially does this correctly as long as you have your site under version control. Although having a DB instance for each developer seems a little bit tricky to me but that may just be my inexperience.

</comment><comment><USER39> What do you think about versioning with an ORM?

Doesn't this imply a automatic versioning with the rest of the code?

For example, I use Sequelize and create my tables via model classes, which I write like every other code in my app.

<comment><USER40> That works until a method you reference in the migration changes or doesn't exist anymore. For instance, a model you're working with might not expose some of the methods you're working with or a library has been modified, but it's being used in the migration. Obviously this is something that people should try to avoid, but I once found myself in the position of having to do checkout to some specific commits in the history in order to be able to run migrations at a past project. In the end I found myself repeating the process once too often and created a gem for it: [https://github.com/hph/farfugl](https://github.com/hph/farfugl). This is for Rails migrations but the concept is the same in any library or language, you just have to go to the commit where the migration originates from.

As a side note, I've been using Sequelize for a year now and it can often be pretty annoying. Both the query interface and the migrations are subpar when compared to Rails. For instance:

        Model.find({ id: 1}) // or maybe just Model.find(1)!

vs. (the Sequelize syntax)

        Model.find({ where: { id: 1 } })

Forgetting the "where" above doesn't even raise an error, it just silently fails.

<comment><USER39> For IDs you can use

        Model.find(1)

</comment></comment></comment><comment><USER41> "every SQL script file must be immutable after it is deployed to production or staging environment."

git doesn't support locking files. Then, what is the best way to meet this immutability requirement?

<comment><USER42> A baseball bat

</comment></comment><comment><USER22> Schema Evolution Manager is excellent for managing Postgres databases in a language-, framework-agnostic manner!

</comment><comment><USER43> SSDT is fantastic for this. I wish it worked for something other than SQL Server though!

<comment><USER44> Well I mean it has "SQL Server" in the name, so..</comment></comment>


IBM z14 mainframe - <AUTHOR> https://www.ibm.com/us-en/marketplace/z14 <comment><USER2> Can recommend this link; it's a one-click victory in buzzword bingo:

> As businesses adapt to capitalize on digital, trust will be the currency > that drives this new economy.

> With z14 you can apply machine learning to your most valuable data to create > deeper insights.

> In the digital economy data is the differentiator. IBM z14 can rapidly > derive actionable insights and enable progressively smarter decisions for > better customer experiences and new revenue streams.

> IBM z14 can help you accelerate development and delivery of secure, scalable > services with new economic models.

With such key features as:

"100% encryption is 100% mainframe"

"Trust – the foundation of digital relationships"

"Facilitate new secure services with Java"

I actually like IBM mainframe tech, but this page is hilarious.

<comment> le-mark The one important part in all the buzzwords:

> New software pricing tied to business value

So, it's like a tax on your business, if you succeed, so do they!

All joking aside, has anyone used mainframes for a greenfield project in the last 20 years? I'm genuinely curious. I've been under the impression the only reason they're still around is because of the massive 20 Million LOC cobol apps hanging around.

I've often thought I'd much rather have a cheap, featureful "walled garden" PAAS for side and small projects, as opposed to wrangling vms and containers all the time. Mainframe could fit this use case.

<comment><USER3> They make some capacity and performance claims when it comes to modern software:

>> it can run the world's largest MongoDB instance with 2.5x faster NodeJS performance compared to paltry x86-based platforms. It supports 2,000,000 Docker containers and 1,000 concurrent NoSQL databases.

<comment><USER4> Don't you use mainframes to have one very powerful machine? Like the central system of a large bank that needs to ensure transactions for millions of accounts book correctly.

If you run VMs anyway, aren't x86 server cheaper? (honest question, don't know much about that)

<comment><USER5> Mainframes used to be a lot about concurrent I/O for transaction processing - e.g. booking/check-in/handling of airline passengers or yes, batch transactions in the millions (e.g. end of day calculation within banks or financial information providers).

TCO might be lower with x86 these days, it all depends on your workload.

</comment></comment></comment></comment></comment><comment><AUTHOR> One of the cool things of machines like these is they have a "presence".

Your generic server sits somewhere on a rack. Your supercomputer is a seemingly endless field of racks.

This one is a large black box that exudes computing power.

I only wish they made the front panels more distinctive between iterations.

<comment><USER6> I wish they sold the case separately ;-)

<comment><AUTHOR> A couple years ago I saw a gutted ES/9000 case for sale. I imagined suspending a Raspberry Pi with metal wires in its middle and let it run Hercules.

</comment></comment></comment><comment><USER7> Fair warning: I am going to ramble a bit.

I kind of wonder. From what I know (which is fairly little, I guess, but bear with me), IBM's mainframe business has profit margins that would even make drug dealers drool. The corollary is, of course, that these machines are very, very expensive.

I wonder what the fincancial pro's and con's would be if IBM tried to loosen up a little on the margin and increase their potential market in return, in other words, sell cheaper mainframes.

I vaguely remember they tried something along those lines during the S/390 era, which culminated in a 4U rack-mountable machine. (I assume, naively, that the smaller physical size and capacity corresponded to a smaller price).

From what I've heard and read over the years, IBM invests a lot of money in its mainframe line, and at least on the hardware side, a lot of really cool technology and research goes into these machines. It's kind of a shame that all this cool technology gets cooped up in relatively small market niche.

<comment><USER8> The simple fact of the matter is that hardly anyone needs a mainframe. Most computing tasks these days can be accomplished better / cheaper with big racks of commodity hardware that you can source from anywhere. In addition the huge price tag (starting at $500k I believe) serves to reinforce the marketing message to the corporations that purchase this stuff that is product is much more advanced, reliable and feature packed than the alternative, and that you want it if you can afford it. Why else would it be so much more expensive, right? IBM has likely done these calculations and figured out that their pricing is optimal for extracting the maximum profit from their customer base.

</comment></comment><comment><USER9> Marketing message is all about 'trust'. Then you see the product images! They clearly forgot to tell their industrial designer what they were going for. It looks like a Terminator's shower stall.

</comment><comment><USER10> I see a lot of people knocking these things in the comments...but if you want guaranteed employment you would be trying to get experience with COBOL. I'm telling you.

Truth is, these things are so much more stable and secure that most projects running on large hardware (think Oracle anything) would end up saving money by running on a mainframe.

<comment><USER11> > Truth is, these things are so much more stable and secure that most projects > running on large hardware (think Oracle anything) would end up saving money > by running on a mainframe.

Stable, maybe, but probably not secure. The software stack on these was designed _well before_ anyone had any real interest in network security. I saw a youtube by a security researcher who took a look at one and found all kinds of inadvisable stuff. He also said there were very few people looking at these from a security perspective.

<comment><USER10> I saw the same presentation. You're right, they could use a penetration team in the release development cycle.

I was referring to the access control model that's built in to Mainframes. They were pioneers of mandatory access control's and fine grained security.

Having fewer people working on them is a definite issue, though, when it comes to the kinds of exposures that Mainframe integration to a corporate environment entail.

</comment></comment></comment><comment><USER3> Who's buying these things?

Compuware has 2,200 customers worldwide. I'd guess most of them have at least one IBM mainframe. They're based in Detroit so I'd say their largest customers are the big three automakers.

eBay was a big IBM customer at one point weren't they? I'm sure there a lot of banks and financial institutions, and big manufacturing companies. Most of the Fortune 1000 then?

<comment><USER12> Banks, government, reservation systems (like airlines) and the odd company that had tech at the seventies.

<comment><USER13> Utility companies as well

</comment></comment></comment><comment><USER14> Disclosure: I work at IBM.

Mainframes are not my area, but what I've learned makes me salivate. The next time I launch a startup, one of these Z systems is going to be on the CapEx plan.

The lowest end tier of these systems is something that could be easily afforded post series-A, and the stunning computational power is almost the least important benefit. These systems provide incredible redundancy, reliability, fault tolerance, and transaction processing speed. Decades of engineering experience have gone into building these.

You _could_ hire engineers to figure out to make systems run reliably on a cloud provider, but unless you are one of a tiny handful of unicorns that specialize in this, you are going to do a second-rate job of it. And hiring the staff with the skill set to do that is incredibly expensive and doesn't create differentiating value.

<comment><USER15> Frankly, when was the _last_ time you launched a startup? Why would you waste money on a technology stack which is essentially the physical manifestation of vendor lock-in and rent seeking, all but guaranteed to grow in cost in lock step with the size of your wallet? It's hard enough to find superlative software engineers and operations staff to work on more mainstream technologies; where are you going to recruit staff proficient in a technology stack alien to most everybody, who also want to shoulder the risk of joining your startup venture?

No matter how much IBM wishes it were not so, we've reached a point where you absolutely _can_ (and should) do your computing on a substantially more open platform -- where more than one vendor can provide each layer of the stack that you build on!

<comment><USER14> > when was the _last_ time you launched a startup?

6 years ago. After that, I was an early (< 10 headcount) engineering hire at a couple companies.

> all but guaranteed to grow in cost in lock step with the size of your > wallet?

Check your premises. My general understanding - again, mainframes are not my area - is that IBM almost guarantees falling per-transaction costs over time with Z.

> a technology stack alien to most everybody

Let IBM maintain the mainframe and let your engineers build on top with whatever tech stack or platform they are comfortable with. IBM invented virtualization in the 60s. And there are quite a few more mainframe engineers out there than you might think.

</comment></comment></comment><comment><USER16> They used to have these easy to understand metrics like 1 mainframe can run 1000 Linux VMs at such and such hardware spec. And that almost made it worth the price. Now I see only blahblah.

</comment><comment><USER17> From a physical design perspective, it really looks like they glued a few supersize nVidia Sheild TV's onto the front of it: [https://www.nvidia.com/en- us/shield/shield-tv/](https://www.nvidia.com/en-us/shield/shield-tv/)

<comment><USER7> Maybe they did. I remember reading that one particular Cray Supercomputer had a MacBook embedded in it just to display a shiny 3D animation.

</comment></comment><comment><USER18> How much do one of these things cost?

<comment><USER3> Bare bones? Probably $200-300,000. Most companies will spend a couple million. And I'm sure if you loaded it to the gills with RAM and disk you could spend $25,000,000 USD.

Sounds like a lot, but it's cheaper than re-writing all the software you're running on your current Z10/Z11/Z12.

<comment><USER19> Do they come with an engineer?

<comment><USER20> _That 's_ when it starts to get expensive.

</comment></comment></comment></comment><comment><USER21> I expect a nice boost in revenues and stock coming up, happens with each z series launch.

</comment><comment><USER22> I wonder about the timing of the release with Q2 earnings report tomorrow...

<comment><USER23> I predict t'will be 21 straight quarters of declining revenues.

<comment><USER24> For the record, balozi earned +1 clairvoyance point.

</comment></comment></comment><comment><USER25> Every time I see something on these mainframes I'm filled with the idle curiosity about what the minimum quanta of a Z system might really be.

<comment><USER26> Hercules emulator on a raspberrypi running a bootleg copy of z/os is about the minimum. Maybe not what you had in mind though.

<comment><USER25> I was thinking more of a minimum reasonable (or perhaps plausible) configuration of actual Z system hardware.

</comment></comment></comment><comment><USER27> How powerful is a Z-series mainframe?

<comment><USER28> Very.

They are also built like a tank, and have so many levels of redudancy and reporting that if any component breaks in your machine they guarantee there won't be any downtime (and a technician is dispatched to fix it automatically). That includes things like CPU cache line bridges, complete cooling failures, storage failures, etc. Also they guarantee that every new version of the mainframe will not increase power consumption (which is absolutely insane).

Oh, and the bill will make you dizzy.

<comment><USER29> They also ship with a small eolic propeller and the case is made with photovoltaic cells. All in case of power shortage. And of course the case contains few pigeons for ip over avian carriers in case of network shortage.

<comment><USER28> I would not be surprised if that were the case. They might even start doing IP-over-technician while they repair the problem with avian flu that spiked the packet loss in the ip-over-carrier-pidgeon channel.

</comment></comment></comment></comment><comment><USER30> Do you still have to interact with it with the worst UI in the history of the world though?

<comment><USER7> As a sysprog/admin? Yes.

I can understand perfectly well why they kept JCL around, what I do not understand is why they have not come up with some successor.

</comment></comment><comment><USER31> Blockchain! Machine learning! Cloud! This is the most Buzzword Compliant mainframe we've ever made!

<comment><USER32> IBM invented Buzzword Compliance. It's probably patented. There is likely an IBM Buzzword Compliance team. I bet they leverage Watson to ensure compliance.

<comment><USER33>
      I bet they leverage Watson to ensure cognitive compliance for strategic solutionings.

Fixed it for you.

<comment><USER34> This just in, after the Buzzword Quality Assurance process:

       I bet they leverage Watson to ensure world-class, mission-critical cognitive compliance
       for strategic cloud-aligned, value-added, blockchain-enabled solutionings.

<comment><USER33> That is horrible and beautiful, well done.

I'm going to go listen to Mission Statement now: [https://www.youtube.com/watch?v=GyV_UG60dD4](https://www.youtube.com/watch?v=GyV_UG60dD4)</comment></comment></comment></comment></comment>


Having a list with ideas, how to identify the good ones? - <AUTHOR>

Hello,
I have a list with 118 ideas. What are good ways to identify the good ones from the bad ones?
I appreciate your answer! <comment><USER3> I create a spreadsheet with ideas down the side and then categorise the columns with different criteria, things like :-

* What price will it be sold for

* How much will it cost me to start up

* Can I do it all myself / is outsourcing needed

* Is it an easy sell / impulse purchase

* How many people have said they would be interested

* How long will it take to get set up

* Does this sound like fun to work on

I would highlight quite positive answers in green, so I can see at a glance with is quite an attractive idea to work on.

I think it's important to then take your own passion & preference in to account. For me I might prefer to pick something with a higher potential revenue with a bigger list of interested people, where as you may put more weight behind something that is short to work on & fun.

</comment><comment><USER4> Try.

* I create a trello board and divide my ideas into a few categories (do-able, I really like, this-is-ridiculous, already-existsi-so-who-am-i-kidding etc)

Then... (maybe)...

* Create a landing page for each one of them (ok so maybe not for all of 118!!!) - <USER2>

* Post on HN, Reddit

* Collect the email addresses & feedback (ask them questions or survey when they sign up)

* Email them back on each of your idea and ask for further feedback (but don't spam)

Above exercise might take you 2-3 weeks. But I find it a practical way of learning - and I think the often more interesting thing is that you learn to "change" your idea depending on the feedback.

</comment><comment><USER5> Check out [http://www.startupbook.net/](http://www.startupbook.net/)

"This book is aimed at developers who want to launch their startup with no outside funding. It's for companies started by real developers solving real pain points using desktop, web and mobile applications."

You'll find various glowing reviews about this book so I won't add to them. The chapter about how to find and select an idea, measure traction etc before you start building anything may be of some interest.

</comment><comment><USER6> SWOT analysis would be a good start.

[http://en.wikipedia.org/wiki/SWOT_analysis](http://en.wikipedia.org/wiki/SWOT_analysis)</comment>


AI the No Bullshit Approach - <AUTHOR> https://blog.piekniewski.info/2020/06/08/ai-the-no-bullshit-approach/ <comment><USER2> Nice article. A bit sad it has received way less attention than hypish criticism of the ML hype itself.</comment>


Why NASA Needs a Programmer Fluent in Fortran and Assembly Language - <AUTHOR> http://www.popularmechanics.com/space/a17991/voyager-1-voyager-2-retiring-engineer/ <comment><USER2> There's still an active community writing tight assembly code for vintage computers like the C64, and there was a ton of excitement around Notch's 0x10c, which was a virtual computer of the same class as Voyager's. There's clearly a yearning among modern programmers for the days where the entire system could be understood by one brain.

This probably isn't practical, or sane, but I wonder what would happen if Nasa released an open source simulator for Voyager and invited the community help implement the software fixes they need. There might some really good devs who would do it just for the challenge.

Think of it as 0x10c with a purpose.

<comment><USER3> The concept sounds cool, but what sort of fixes could the Voyager probes possibly need? They were launched in 1977, that's almost 40 years ago. As far as I know they're waking up every once in a while and sending telemetry back.

It would be cool to have access to the source code and an emulator, but I really don't find it plausible that members of the general public could submit patches to something that's been running smoothly for 40 years, and where the risks of changes far outweigh the benefits of keeping the existing software running as-is.

<comment><USER4> The article says that part of the need to get programmers back in the project is to be able to get exact readings on the amount of energy that's being used and the amount left.

It appears like they are currently working with estimates (e.g. they cite a use of 3.2 Watts but it's actually 3.0 due to margins of error in the original design. Now they want to know exactly how much power is consumed).

So maybe they don't need fixes in the traditional sense that something is broken, but fixes in that they need the spacecraft to do stuff that wasn't designed for.

Another example they cite is when they revamped the software in 1990 so they could better automate certain functions since they were preparing for the interstellar part of the journey and they needed certain automatic procedures which apparently were done manually in the past.

<comment><USER5> It seems being a fortran/assembly programmer is just a small part of whats needed.

What they seem to need are engineers that can teach themselves quite deep domain knowledge of the voyagers to figure out how they work, reverse engineer the hardware and software that arn't documented. (and eventually do some programming, mainly on old ground based code, and certainly without bricking the probes if the flight software needs to be altered).

</comment></comment></comment></comment><comment><USER6> PopMech makes it sound more arcane thane it might be: it's not _that_ much out of the ordinary to know (or learn) assembly and Fortran or to have to work with computers that have 64K RAM (some _modern_ microcontrollers have less[1]) I imagine it would be as hard as it sounds if they would require a specialist with years of experience in their particular dialect of Fortran and some particular assembler, expecting them to be productive right away. But it looks to me that they just have a peculiar setup and need someone who is able and willing to learn it - that's a situation plenty of companies have.

Now a Voyager computer itself looks like a peculiar piece of technology in itself - is it one of a kind? If so, who made an assembler and Fortran compiler for it? Unfortunately, the article (or any others I googled up) doesn't go into that. Is there a description of Voyager computer publicly available?

So far, I find it the most marvelous that a tape recorder is alive and well after 40 years of regular usage: the computer memory of the day was not nearly enough to hold scientific data, so it's being written on a 8-track tape recorder, sent to Earth and then overwritten again, in a loop. Still, even with minimial usage, I have trouble imagining how a single tape reel could last 40 years of regular use. In space.

[1] List of common microcontrollers: [https://en.wikipedia.org/wiki/List_of_common_microcontroller...](https://en.wikipedia.org/wiki/List_of_common_microcontrollers)

STM32 F0: [http://www.st.com/stm32f0](http://www.st.com/stm32f0)

</comment><comment><USER7> I am 36, coding 3h a day in Fortran for the oil and gas industry. I must say, even if I write a lot of Python code too, I really enjoy the way a program is "optimized" by default, that is, if your algorithm is good, you do not need to do crazy things to get good performance out of your hardware.

<comment><USER8> Care to share any references on learning modern Fortran? I've tangled with some pretty ancient code (full of Hollerith constants and tangled mazes of GOTOs) but I get the impression there's a whole other world of Fortran out there.

<comment><USER9> I've found Metcalfe, Reid, and Cohen[1] to be a fairly clear refresher on Fortran. Unfortunately, the build tools I use don't support many of the nicer features like user-defined types.

[1] [https://books.google.ca/books/about/Modern_Fortran_Explained...](https://books.google.ca/books/about/Modern_Fortran_Explained.html?id=MmOIFEgsDLwC&redir_esc=y)

<comment><USER8> Thanks for the reference! I've put it on my reading list.

</comment></comment></comment></comment><comment><USER10> Fortran is widely used for scientific computing and in high performance computing some knowledge of assembly language is useful in making use of vector instructions - compiler auto-vectorisation still needs a lot of help.

Fortran and assembly language are key skills at every national supercomputing center in the world.

<comment><USER11> Yea, but embedded Fortran, and assembly of whatever architecture that voyager runs on is a different beast.

<comment><USER12> Point well taken. Couple that with the need for an intimate knowledge of idiosyncrasies endemic to salvaged 8086 chips operating in space.

</comment></comment></comment><comment><USER13> I use Fortran code which was written before I was born on a daily basis. It works exactly as you expect.

I can't get C code to behave the same from one release of the compiler to the next.

Thank goodness NASA uses Fortran.

</comment><comment><USER14> I don't get why they need to hire someone that knows Fortran already. I'm sure any competent and versatile programmer would be able to learn quickly those technologies.

<comment><USER15> That is not how we do any more. You have to have the skill. That is why you have a college degree. You should have total knowledge of everything. You must be proficient too.

</comment></comment><comment><USER16> Given that the Voyager probes are pushing the absolute edge of humanity's reach into the cosmos, I'd love to see NASA publish the hardware specs and open up a competition to find the best possible solution.

With open specs, a simulator or emulator could be built, and anyone with interest could give it a try. A core team of NASA engineers might come up with some unit tests and acceptance tests. If these are passed, then the code is eligible for professional review.

</comment><comment><USER17> It's going to be hard to find a person with the right set of skills regardless of their age. One of the skills required is patience and another is absolute dedication to getting it right first time. Not a job for a Scrum enthusiast.

<comment><USER18> Yah, I'm not sure the assembly is the hard part (I know a couple engineers in their 20's that do just fine with assembly), or for that matter layering fortran on said engineers. The problem is that none of them will likely be interested in working on a piece of hardware for 10 years that effectively is a dead end. Its the same decision that I looked a couple years ago with regard to mainframe development. Its a dead end and the longer you stay there the less employable you will be for your next job.

AKA, if whoever gets the job at NASA is younger than the 50 Y/O's they are looking at, it should come with a pension, because getting the next job will be hard.

<comment><USER19> > Its a dead end and the longer you stay there the less employable you will be > for your next job.

I don't understand why people have this mentality, because it's so wrong. We live in an era were kids can get jobs fresh out of high school after taking an eight-week crash course in programming an no other experience. Certainly a person capable of building robust software for NASA is capable of learning whatever platforms exist in 10 years.

I've never had two jobs with an overlap in work. I've done everything from embedded development to data science and analytics. Being an expert in many fields demonstrates that know how to become an expert.

<comment><USER20> Seriously.

Who the fuck is going to look at a resume with "last 10 years at NASA building shit for a MOTHERFUCKING DEEP SPACE PROBE" and go... nah, dude(tte) doesn't have node.js on his resume... NEXT!

<comment><USER21> Honestly, I have spoken to and worked with some of these engineers, and if they've been ten years deep on software that has no capacity for errors at all, they can be a bit difficult. Not just "I can't use that tool, it has a garbage collector", but when they are put in front of a frontend, they end up re-inventing basic things constantly, because they have absolutely no context.

Honestly, going from that kind of development to web development, "last 10 years" doesn't mean much -- they are essentially junior programmers again. Expect them to break things because they self-closed a script tag. Except them to break things because they forgot to set the button type to button and so some browsers POST. Heaven forbid you use tech where it doesn't default HTML and SQL escape. It's very very easy to forget the amount of extremely specific domain knowledge we all have that keeps things from imploding, and if they don't have it, their code doesn't hit production without a senior dev looking over it.

<comment><USER20> You hire an extremely smart, talented individual with some ramp up time, or pass on them for someone less talented (and less smart?) with less ramp up time.

I can't speak to your current needs when hiring someone, but you're mistaken if you think I was assuming they can pick up web dev with no mistakes made.

</comment></comment></comment></comment></comment></comment><comment><USER22> Previously: [https://news.ycombinator.com/item?id=10476306](https://news.ycombinator.com/item?id=10476306)

I really, really wish HN would be fair and go with the first post of a story, not the creatively-titled later submission.

</comment><comment><USER23> Some computational physics courses in parts of Canada are still taught in Fortran90. I also worked in a lab (in 2005) which still used 5 1/4" floppy disks. The difference between these and NASA, of course, being choice.

<comment><USER24> Yes. A good friend of mine still uses FORTRAN regularly for scientific computing, and has worked at supercomputing centers all over the world (NCSA, ORNL, IVEC in Australia, LBNL...). She just got her PhD around 2006-2007-ish, so it's not like she's ancient, either.

</comment></comment><comment><USER25> crowdsource it. there are thousands of programmers that would love to help, let alone audit the result. and even more that would love to write the sims.

</comment><comment><USER26> Truly fascinating read. Thanks for posting that!

</comment><comment><USER27> Ok so where's the Job listing would be interested how good a match it would be for me :-)

<comment><USER27> Down voated for this really?</comment></comment>


Ask HN: Have you used any cool apps built on the Steam API? - <AUTHOR>

<comment><USER2> I know the guy that made <http://tf2recipes.com/> \-- and it's a pretty nifty iOS app that makes use of the Steam API.

</comment><comment><AUTHOR> would love an intro (email@example.com)</comment>


Interview with Twitter Co-Founders Evan Williams and Biz Stone - <AUTHOR> http://kara.allthingsd.com/20090414/twitters-co-founders-evan-williams-and-biz-stone-speak/ <comment><USER2> Interesting interview, but not as good as <http://www.charlierose.com/view/interview/10118></comment>


Website review - <AUTHOR> http://flyamber.com.au/ <comment><AUTHOR> Hi Would like some feedback on my site about the design, quality, if you would purchase from the site or not. Thanks!</comment>


Tell HN: So suddenly HN is fixed size and way too small? - <AUTHOR> Not sure about other people,but today the zoom suddenly doesn't work anymore. And the default size is way too small. Hope someone from the devs notices this. <comment><USER2> Working fine for me on mobile safari. Sometimes someone will post a long headline that doesn't wrap, which makes the list pages miserable to browse until you click "hide" under the offending post.

</comment><comment><USER3> > _Please don 't post on HN to ask or tell us something. Instead, please send > it to firstname.lastname@example.com_

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

</comment><comment><USER4> (Browser) zoom works for me. What browser are you using?</comment>


Christians Performed Gay Marriages 1000 Years Ago - <AUTHOR> http://gilesbowkett.blogspot.com/2012/05/christians-performed-gay-marriages-1000.html

<comment><USER2> I suppose if we are going to bring up things they performed 1000 years ago, we can't forget to mention all of the horrible things as well.

Slavery was a normal part of life. Does this mean it's okay too?</comment>


Some Lessons From the First 24 Years of Life - <AUTHOR> http://designcodelearn.com/blog/2012/10/30/24-years-of-life/

<comment><USER2> "When I was a boy of 14, my father was so ignorant I could hardly stand to have the old man around. But when I got to be 21, I was astonished at how much the old man had learned in seven years." -- Mark Twain

</comment><comment><USER3> As a 38 y/old: You have no idea _how much time_ you have right now. I thought I was always really busy, but looking back I didn't realize how much time I actually had when I was in my 20's. Try kids, your own house, (older) relatives, everyday stuff, a body that requires more 'maintenance',a full-time job and a startup. OP says "money is fuel", but _time_ is perhaps even more so.

<comment><USER4> Oddly enough, at 35, I'm very much aware of how much time I have. I have the full-time job, but virtually none of the rest of the time-expenditures that someone my age usually has. It's very much like me at 20.

And yet I'm acutely more aware of my time and how I spend it.

<comment><USER3> I wonder if this holds up: The older we get, the more aware of time? Or is it: The _more responsibilities_ , the more aware of time?

</comment></comment></comment><comment><USER5> As a 31-year old, the main thing I've learnt is that at 24, I knew nothing. Nothing at all.

I imagine I will feel the same way about 31-year-old me, who is at last, _finally_ , starting to Get. It.

Conversations with my 69 year old Dad suggest that he too feels as though he is on the cusp of true understanding.

Also, here's a joke for old farts.

        When I was 18, I actually did know everything*

        * for sufficiently small values of 'everything'

<comment><USER6> This is exactly what I thought when reading the article. "Wow, this kid has a lot to learn."

And I would hope I'd say the same about myself in ten years. And ten years after that, etc.

<comment><AUTHOR> I think one should worry if one ever feels they've learned all they need to know. At the same time some 30 or 40 year olds will say things like "just wait until x happens..." and I always wonder why they stop there and don't get into more detail. I guess at 14 most of us wouldn't have grasped theses "truths" which reveal themselves later. Maybe we can't at 24, 44, etc. I'm really interested in what others feel I will learn not because its true (it may be) but because I like to see if our perspectives differ, and why.</comment></comment></comment>


Good enough for government work? The contractors building Obamacare - <AUTHOR> http://reporting.sunlightfoundation.com/2013/aca-contractors/

<comment><USER2> Basically bullshit when you factor in that the one contractor not hired was the integrator. HHS's Centers for Medicare and Medicaid Services (CMMS), i.e. inexperienced/no experience in projects of this scale inexplicably decided to be the integrator of this wildly spread out project, as the report details.

And did a horrible job of it, as we're hearing from the NYT " _In the last 10 months alone, government documents show, officials modified hardware and software requirements for the exchange seven times._ " ([https://news.ycombinator.com/item?id=6583327](https://news.ycombinator.com/item?id=6583327)) and from many sources that changes were being required though the last week before the launch. Which of course had to be coupled with only trying to test the whole thing in that last week.

Now matter how evil you believe these contractors to be, no one can realistically claim they could have produced a working system under these constraints, including how late they were brought into the game.</comment>


SeaMicro drops an atom bomb on the server industry - <AUTHOR> http://venturebeat.com/2010/06/13/seamicro-drops-an-atom-bomb-on-the-server-industry/

<comment><USER3> Performance per watt depends on the computer _and the workload._ It's not apparent from the SPECint benchmarks they show in the article, but chips like Atom are better than beefy server chips for some server workloads, and worse for others, when measuring performance per watt. Here's a paper from Berkeley's RAD lab which tries two Atom processors and a Xeon on several different server workloads, and compares their performance per watt:

[http://www.sigops.org/sosp/sosp09/papers/hotpower_10_chun.pd...](http://www.sigops.org/sosp/sosp09/papers/hotpower_10_chun.pdf)

The tl;dr version is that what you really want are _hybrid_ datacenters, where you can assign various workloads to different types of machines, and use each machine type for what it's best suited to do.

<comment><USER4> See also this:

[http://www.realworldtech.com/page.cfm?ArticleID=RWT090909050...](http://www.realworldtech.com/page.cfm?ArticleID=RWT090909050230)

for a comparison of various types of chips.

<comment><USER3> That chart compares theoretical peak GFLOPS per watt, which is an absolutely terrible metric for server workloads. As the paper I linked to showed, Atom is sometimes much better and sometimes much worse than large, fast server processors in performance per watt _on real server workloads._ Those charts don't take into account things like cache, or branch prediction, or multicore memory coherence.

The only way to get meaningful performance per watt data is to actually run real workloads on various processors.

</comment></comment></comment><comment><USER5> I worked with one of these prototype boxes. I was using it for something a bit outside their common use case: clustered ETL processing of log data. I was quite happy with the performance. In the workloads I had that needed lots of threads, I was able to use the box to spin up a _lot_ of nodes and crunch through several hundred GB of log data very quickly. The machines were easy to work with since they felt like normal Linux nodes, and the interconnect fabric made inter-process communication very snappy.

<comment><USER6> I'm still confused, is it a single multiprocessor machine, or is it a 'cluster' in the sense that each node has a single processor? (You used both threads and inter-process, but also implied multiple nodes...)

<comment><USER7> >I'm still confused, is it a single multiprocessor machine

I would love to watch that thing boot, haha. SO many penguins!

<comment><USER7> I apologize for the self-reply, but am I wrong here? I use mostly OpenBSD and linux only lives on servers that I don't physically see booting. I remember when I first starting using Linux, there would be one penguin for each processor core. Booting it up for the first time on a dual xeon machine and seeing 8 of them, was kindof funny in a nerdy way.

Would this be different? If there are 512 cores...would it show a penguin for each core?

Sorry, like I said, I haven't actually watched the boot process on a linux machine for quite some time, and now I'm a bit curious...

Sorry if this question is juvenile or stupid or something.

<comment><USER8> You're right; IIRC some SGIs show a whole screenful of penguins. But you're kind of derailing the conversation.

</comment></comment></comment></comment></comment><comment><USER9> James Hamilton, VP at Amazon and author of the popular blog "Perspectives" has some interesting insights into and great things to say about the work SeaMicro and other new startups are doing to revolutionize the server industry.

[http://perspectives.mvdirona.com/2010/06/14/SeaMicroReleases...](http://perspectives.mvdirona.com/2010/06/14/SeaMicroReleasesInnovativeIntelAtomServer.aspx)

<comment><USER10> See <http://news.ycombinator.com/item?id=1429625> for HN item on this excellent posting of his with a few comments, e.g. I note he notes this system has no ECC. And that Intel is pretty unique in that limitation.

</comment></comment><comment><USER11> "People who are really serious about software should make their own hardware." - Alan Kay

This news is yet another data point that developers will need to hack concurrency sooner than later, as a core skill in one's professional repertoire. Off to learn Stackless PyPy, Clojure, Scala, etc...

<comment><USER12> I think I can afford to wait until people start proving undeniably (through out-competing rivals) that Scala and Clojure have any pragmatic benefit in real multicore programming tasks. Anything that mentions Fibonacci numbers doesn't count, nor do easily parallelizable problems commonly handled in C/C++ (ray tracing, media encoding, web servers, etc.) It needs to be something not trivially broken up into independent parts.

<comment><USER13> While this isn't undeniable proof, it's interesting none the less, Simon Peyton Jones: Data Parallel Haskell.

<http://www.youtube.com/watch?v=NWSZ4c9yqW8>

He talks about how this scheme can parallelize non-trivial data structures, and how tedious book keeping can be avoided. I haven't used it yet in a project, so I can't speak of its efficacy - but worth a watch, imho.

</comment></comment></comment><comment><USER14> Their tech overview PDF has less layman fluff than the article: [http://dev.seamicro.com/sites/default/files/SeaMicroTechOver...](http://dev.seamicro.com/sites/default/files/SeaMicroTechOverview.pdf)

</comment><comment><USER15> Interesting in terms of technology, but of no interest to me as someone who does colocation and web servers for my clients, especially since they almost all use traditional RDBMSes like MySQL and Postgres.

The Atom is too underpowered and too RAM limited for individual systems - <USER2> would do better in most cases with a 2x quadcore setup and 32-64GB RAM combined with OpenVZ or Solaris Zones. Lack of ECC = automatic disqualification for me as well.

For a company that is doing a lot of web serving a la Facebook or eBay I can definitely see the appeal. In such larger cases, power usage dwarfs many other considerations.

<comment><USER16> I would rethink that if I were you. What if you started to provide virtual private servers on one of these boxes instead of offering colocation? I think this innovation makes VPS solutions even more efficient than they are today.

<comment><USER17> ECC is a huge thing in that case.

The thing about being a VPS provider is that if your hardware chokes, your customers notice, and they get pissed. Sure, if you are running some kind of web farm you set it up so that things keep running without interruption if you lose a server. But if you provide VPS hosting and you lose a server? People notice, and they will be pissed. If you are a VPS provider and you loose customer data? they will parade your severed head around town on the end of a pike.

Amazon has successfully tackled this problem by training their customers that they can shoot a server at any time. This is good for them, but it's not how VPSs have traditionally operated. customers expect the thing to stay up and for their data to stay safe.

<comment><USER18> I've seen ECC mentioned a few times in this thread but how relevant is it really? At my old job we used both regular COTS hardware from Dell and HP and high-performance gear from SUN, often at four times the price. The latter had ECC memory and, if memory serves, shielded CPUs but I never noticed much difference in stability or performance when compared to regular hardware. If someone has some numbers on this, I'd love to see them.

<comment><USER17> if your dell and HP kit was server grade, it almost certainly had ECC ram. before nehalam, it was almost impossible to get dual-cpu servers that would support non-ecc ram. Even today, now that unbuffered ram is supported (buffering and ECC are not the same thing, but it's unusual to see registered or buffered ram that is not also ECC) nearly all server hardware you get from the likes of dell or HP is going to default to using ECC ram.

The big deal with ECC is that with non-ecc ram, you don't know when in-ram data was corrupted. If you have a bad bit of ram that is being used for our journaling/disk caching, it's pretty easy to corrupt your data.

If you've got ECC, it corrects single bit errors, and logs that there was a problem, and it can be configured to crash on double bit errors, rather than just silently corrupting your data.

If you use non-ecc ram, not only will bad ram silently corrupt your data, and sometimes crash your server, but there will be no indicator (other than random crashes) that the problem is in fact bad ram.

You can get away with not having ecc in large server farms if you have lots of checksums on generated data... but in a VPS type situation where people get pissed if you loose a single server, it's sure to end in tears.

</comment></comment></comment></comment></comment><comment><USER19> I hope this shakes up the VPS hosting industry, too. I'd love to see what kind of Linode or EC2 style on-demand computing these might provide.

<comment><USER20> Hopefully it will be cheaper. $20/month might sound very cheap but for some guys (like myself) living in 3rd world countries, it's almost next to impossible.

<comment><USER17> Hey, I have a question. If I wanted to advertise to /technical/ people in your area, where would I do it?

Note: I'm not really interested in business people; I want *NIX enthusiasts who are /not/ spammers, who can write English and are capable of running a Linux box that won't get compromised, and who don't need much help from me to run the thing.

<comment><USER21> Why not just buy google ads for technical keywords and target only the desired countries?

<comment><USER17> For my ad money, I like display ads rather than pay per click ads... I am just trying to say 'Hey, I exist' and pay per click is usually not a very cost- effective way of doing that.

(now, most of my advertising 'budget' was spent on, say, writing a book... but per-click ads are the opposite of what I want.)

</comment></comment></comment></comment></comment><comment><USER22> Interesting... I've never been in a really big datacenter, so I'd like to see some (hopefully non-biased) reviews from somebody that does work in those places.

Would this really work well for the intended market? There are lots of startups over here that plan on massively serving webpages - would something like this (only cheaper :) ) make you reconsider using whatever cloud services you're currently using?

Articles I found on Google:

[http://gigaom.com/2010/06/13/seamicros-low-power-server- fina...](http://gigaom.com/2010/06/13/seamicros-low-power-server-finally- launches/)

and the Wall Street Journal's take:

[http://blogs.wsj.com/digits/2010/06/14/seamicro-tries-to- ret...](http://blogs.wsj.com/digits/2010/06/14/seamicro-tries-to-rethink-the- internet-server/)

<comment><USER23> _There are lots of startups over here that plan on massively serving webpages_

Seamicro is using the same sort of lateral move that RISC made and that Transmeta attempted to apply: Could certain functions happen more efficiently if moved outside of the traditional 'box?'

Could a datacenter provide a low-latency infrastructure for front-end web- servers while reducing their power and other expenses? I think there's a good chance this is already being done.

</comment></comment><comment><USER24> I do see a potential problem here. In the pictures, they show a bunch of Atom CPUs soldered directly to the board. That means dire things for service. Now, if a single CPU has a a flaw, you need to replace an entire board of CPUs.

Compare this to a standard blade setup, where you could just swap out CPUs, or even an IBM System Z where you could hotswap the CPU, and service doesn't look so great.

<comment><USER8> To be fair, the unit of hotswap in System z is a book that contains ~32 cores; that's not so different from SeaMicro.

</comment></comment><comment><USER23> The way I read this, they are achieving savings by virtually mux-ing (or de- muxing depending on viewpoint) much of everything that's not the CPU. Is this optimized to make the support of virtual servers with relatively low throughput more efficient?

<comment><USER25> I read it to be they created a chip that runs code written to emulate/virtualize most of the other (non-cpu) logic on a motherboard. so instead of 10 "io" chips and 10 "memory" chips most of which are idle they have 15 virt chips which may act as either "io" or "memory" depending on demand.

I guess (de)muxing is close to that? Multiplexing many "request" amongst fewer real(non-virutal) objects.

<comment><USER23> _I read it to be they created a chip that runs code written to emulate/virtualize most of the other (non-cpu) logic on a motherboard._

That's exactly what I meant by "virtually (de)muxing everything but the CPU." I should have said, "CPU+chipset," though.

I wonder if their architecture is also (unintentionally) optimized for languages like Ruby/Python? I suspect that languages like those tend to have more CPU operations per IO operation. I wonder if anyone has researched this metric?

</comment></comment></comment><comment><USER26> This seems quite similar to the FAWN project at CMU. <http://www.cs.cmu.edu/~fawnproj/> The idea is similar: if IO is the bottleneck, instead of scaling up IO, scale down the CPU power.

</comment><comment><USER27> It would have been better to see a comparison with SGI (ex-Rackable) CloudRack systems, which have a bit of an inbetween approach, using Xeons, but at least nominally seem to pack in more cores in the same enclosure size. One of their power tricks, in addition to pulling back DC converters further from the computers, is to allow things to run hot, resulting in savings on cooling costs.

<comment><USER28> The SGI machines are geared pretty heavily toward supercomputing applications. As such they're tailored for applications that need to share data between processes efficiently, and that also need to do a lot of computing.

The SeaMicro server probably would look pretty lousy when compared on SGI's turf, since Atom processors aren't designed to be performance-competitive with the current desktop and server processors.

What would make more sense is a comparison a datacenter for a company like Disney or Google or Amazon. I suspect that even though SeaMicro's rig would look pretty bad for heavy number-crunching apps, it would most likely do very well in web and database. Those apps don't need to share a lot of data between threads/processes, and they're generally I/O bound rather than compute bound. With proper caching setups, they're mostly network bound, rather than disk bound also.

<comment><USER27> I was thinking of the Rackable stuff, not the supercomputing stuff. They actually acquired SGI, but took the SGI name. [http://www.sgi.com/company_info/newsroom/press_releases/2009...](http://www.sgi.com/company_info/newsroom/press_releases/2009/april/rackable.html)

<comment><USER28> I'd forgotten that Rackable did they acquiring in that one. Oops :)

</comment></comment></comment></comment><comment><USER29> Great news! Hardware innovation typically means new software opportunities. If this is turns out to be a generally accepted workhorse server design and not just a hotrod box I wonder who will be the first in here to develop a profitable software product for it.

Dell products have features and services that make them enterprise friendly, they are more than just hardware to the customer. So trying to compete with Dell head on might not be the company's best strategy at first. Perhaps following the strategy EMC used with CLARiiON to sell through DELL would be more of a money maker for the company.

</comment><comment><USER30> [http://en.community.dell.com/dell- blogs/b/direct2dell/archiv...](http://en.community.dell.com/dell- blogs/b/direct2dell/archive/2009/05/19/dell-launches-quot-fortuna-quot-via- nano-based-server-for-hyperscale-customers.aspx)

Similar concept from Dell, from over a year ago. Although Fortuna seems bit more conventional, I'm not sure if it's a bad thing.

</comment><comment><USER31> Anyone care to suggest some ideas on a few things:

a) who would likely buy these (corporates, SMEs, startups)?

b) it seems that they have increased the risk of single point of failure (e.g. 1 PSU taking down 128 nodes) what's the mitigation strategy?

c) what would an architecture on a box like this look like? Should I just be thinking of it as a cheap set of VPS nodes?

d) People keep mentioning the kind of processing these chips are good for and not so good for. Can someone be explicit about good real world uses and bad ones?

<comment><USER8> a) Web 2.0 sites.

b) It has four redundant power supplies, as do all systems of this size.

c) It's a Beowulf cluster.

d) See the FAWN, Amdahl blade, Gordon, and CEMS papers.

</comment></comment><comment><USER32> I would think that ARM chips would use less power than Atom; but compatibility with existing software is a big selling point (as always). But this would provide a pressure towards ARM servers, and therefore ARM-compatible software.

OTOH, I get the impression that the bulk of the power savings are not from the CPU at all, but from virtualizing the other components. Therefore, the pressure towards ARM is much less.

And I suppose Intel wouldn't be supportive unless there was a compelling long- term reason to choose Atom.

</comment><comment><USER33> Does this really solve a problem anyone has though?

It seems we have an oversupply of CPU on modern boxes and an under supply of I/O speed (or space with SSDs) and memory.

</comment><comment><USER34> Interesting article, but a terrible title! Atom bombs, really?

<comment><USER35> I think it's supposed to be a pun...

<comment><USER34> Oh dear, I did not realize that. Internets one, wingo zero...

</comment></comment></comment><comment><USER36> With that many processors crammed into such a small case isn't heat dissipation a problem? Or are the Atom chips used not as power hungry as a standard commercial cpu?

<comment><USER37> "Atom chips used not as power hungry as a standard commercial cpu" is quite literally the entire point of this startup/product and the article. You had to quit reading before the second sentence to miss this:

> The startup is announcing today it has created a server with 512 Intel Atom > chips that gets supercomputer performance but uses 75 percent less power and > space than current servers.

</comment></comment><comment><USER38> Where can I invest?

</comment><comment><USER39> I am completely baffled that Intel did not do this. I guess they figured ATOM is just a crappy little side-project processor, not one of the big boys.

<comment><USER40> Some see the move to many-core ATOM servers , as a move to put financial pressure on intel(since ATOMs are cheap).

And this move helps the future of ARM based servers , since ARM is more similar to ATOM than to XEON.

<comment><USER10> There's going to be limits to the amount of pressure as long as Intel doesn't offer ECC for any Atom variety microarchitecture. One of the articles I read said that they thought about using an AMD chip but it wasn't going to ship soon enough. I can see them hearing from their customers (would be and actual) that they'd buy more if they offered ECC and SeaMicro offering an AMD based box in the future.

I wonder how Atom specific the ASIC is....

<comment><USER40> i think that the chip also suits the ARM architecture without changes.

</comment></comment></comment></comment><comment><USER41> There are no information about the chipset, memory specs and extension slots in particular.

It could be a huge bottleneck between CPUs and RAM, because of concurrent access of so many CPUs to the memory while Atoms have a very small caches.

It means that real-world applications, like a multi-threaded services (especially JVM-based, or simply MySQL) could not be used efficiently.

I'm also very skeptical about hundreds of KVMs which is a the only working virtualization technology I know. =)

<comment><USER10> Here's a lot more technical detail: [http://www.anandtech.com/show/3768/seamicro-announces- sm1000...](http://www.anandtech.com/show/3768/seamicro-announces- sm10000-server-with-512-atom-cpus-and-low-power-consumption/2)

Per CPU (8 per board):

      Atom Z530 (1.6GHz, Silverthorne)
      US15 chipset (Poulsbo)
      2GB DDR2

Per board:

      4 ASICs to virtualize IO
      2x PCIe x16 connectors

" _Each hop takes 8 microseconds_ ", so good enough for (most/all?) IO but not good enough for memory.

No ECC of course, it being a non-server Intel chip. Each single cored CPU has its own DIMM.

 __* __* __* __* EDITED to clarify the layout per CPU and per board after c00p3r made it clear I wasn't being clear. __* __* __* __* __ _

<comment><USER41> So, it's just a tight packing of a 512 netbook's boards in a one box. I thought it is a 512-core system. Oh, I'm such a moron. Nothing to see here.

<comment><USER10> No, no: each board has 8 CPU/chipet/DIMM(s?) (latter on the other side of the board) with 4 ASICs each handing all the rest of a motherboard's IO, virtualizing it (e.g. each core sees 4 virtual SATA drives). That's why each board needs 2 x16 PCIe connections, the backplane has a fancy supercomputer style torus topology interconnect, and you can put up to 64 physical disks in front. Lots of Ethernet out the back and there's an FPGA system for the backplane, which might be due to the small number of units but will give them lots of flexibility.

</comment></comment></comment></comment><comment><USER42> An ARM-powered variant would sit on top of this Atom-powered machine, with quite possibly the same or higher numbers of efficiency when looking at performance-per-watt.

The article quotes the interviewee saying that ARMs can be used, but not clear enough to determine if they actually have ARM versions of these machines, which I'd be very interested in seeing 100k specint_rate figures for.

<comment><USER3> Most of their technology is not related to the processors themselves, but to the rest of the computer. They've got a fancy high-bandwidth, low-latency interconnect between the chips. They multiplex access to I/O hardware, which I'm sure takes advantage of that nice interconnect. They have hardware- accelerated network load balancing which tries to keep the CPUs at maximally- efficient levels of utilization, letting the spare CPUs sleep, and avoiding malfunctioning CPUs. I would love to see this with ARM Cortex-A9 processors, and I think that's actually very likely if this company succeeds.

</comment></comment><comment><USER43> Big surprise, Dell, Hewlett-Packard and IBM not innovating? I can't believe it...

Just look at the computers they are selling. most still offers 512M of RAM as a default. What can you run on 512? Nothing like having to upgrade the day you receive your product. Consumers by and large need the guidance of manufacturers to make the right HW choices and the manufacturers just want the cash. They will suck each segment dry until the market forces them to make the changes. That's progress?

With all the advances in multi-monitor add-on SW or the fact that Windows and Mac OS have been supporting multi-monitor control for years, try finding hardware already fitted with multiple display connections. In the end, the user is forced to customize their own equipment. And yes, I know most HN readers do this, but I am talking about the general public.

Great to see the little guys are still fighting. The big ones really don't give a damn.

<comment><USER44> > What the F* can you run on 512?

A lot, really. I have rendered DVD quality movies, handled product composition databases and processed millions of orders on machines with less than 512 megs of memory.

Not many companies (or people) have enough data or volume to stress a machine with 512 megs of RAM. I agree it seems a pittance - my netbook has more than that - but if I break up memory usage by application here, Firefox takes most of the RAM, then Gwibber, then Rhythmbox. None of them would be running on a server.

It would be interesting to write down a list of what you can't do with 512M.

<comment><USER43> Well, you can't effectively run Mac OS X or W7 on 512 to start, the most used operating systems in the world. Even Ubuntu needs an upgraded graphics card and RAM to utilize all of the bells and whistles.

As I said before, I was referring to the general public. There will always be hackers or high-end users that can make miracles.

<comment><USER45> I think the general miscommunication here is that you seem to be talking about desktop computing, while this article is focused on datacenter usage.

<comment><USER43> I agreed with the article in the sense that a smaller company has seen the error of some major manufacturers' ways. My opinion was that these major manufacturers do not have the user's best interest in mind and used their PC sales as another example.

They have billions for research at their disposal and they couldn't see that their equipment was too power hungry, would not be sustainable in the long run and probably needed re-tuning? I'm guessing they did.</comment></comment></comment></comment></comment>


Ask HN: Funding situation – help and inputs please - <AUTHOR> Product and competition:
Me and my cofounder have been working on a SaaS product since last year. Then a lightning struck - we pivoted big time. And put our heads down to work. Now we are way ahead (5-10x) than competitors who are funded ($9+ million).
Our funding situation:
We are in stealth with no funding (not much runway).
If we release our product - competitor will know what they lack
and they can put more resources to catchup. We lack ammunition (team and funding) to compete with them.
Question:
What would be a good path to bring product to market ?
Should we approach friends-and-family or 2nd/3rd level contacts or traditional investors ? <comment><USER2> Find a customer, any customer. If you're light years ahead of the competition, they'll say so.

If you built something you can't sell, then it's not worth anything to the market.

Don't worry about your superior competition getting the jump on your technology. They'll do fine without and they know it (and it's your uphill battle to prove them wrong). They're superior because they sell more than you do, and that part of the operation is an essential part it appears you have overlooked.

If you thought selling was hard, try raising money.

Definitely shake down your friends and family if they've got money to speculate with, as they tend to offer ridiculously friendly terms. Traditional investors will leverage your limited runway against you, so don't waste your time with them until your circumstances require you devote your full time to hitting them up.

The thing that impresses investors most is knowing you don't need their money. The way to get there? Sell your product and get your customers to love you.

<comment><USER3> Yep forget about all that stealth nonsense. No customers, no business.</comment></comment>


Greece's Public Sector merges its data, how it was done - <AUTHOR> http://fuzz-box.blogspot.com/2011/06/greeces-public-sector-merges-data-how.html

<comment><AUTHOR> Kallikratis is the codename of the largest project ever conceived in the later years (2010-2011) of Greece's public sector digitalisation / computerisation effort and was founded by the Ministry of Interior and Decentralization.

The goal? To migrate different datasets spread among different applications and databases withing the public sector's infrastructure.</comment>