Make the Type System Do the Work - <AUTHOR> http://nathan.ca/2014/02/type-rich-programming/ <comment><USER2> This article uses _inheritance_ in C++ to ensure the types line-up, but I have really taken a shine to _algebraic data types_ for my firm's data feed handlers.

Here is an example in C++11 that implements a market-data spec line-for-line.

      #pragma pack(push,1)

      struct Quote {
        char     symbol[16];
        uint16_t bid;
        uint32_t bidsize;
        uint16_t ask
        uint32_t asksize;
      };

      static_assert(sizeof(Quote) == 28, "Quote size wrong");

      struct Trade {
        char     symbol[16];
        uint16_t price;
        uint32_t size;
      };

      static_assert(sizeof(Trade) == 22, "Trade size wrong");

      struct Data {
        enum class DataType : char {
          kTrade = 't',
          kQuote = 'q'
        };

        uint64_t sequence_number;
        DataType data_type;
        union {
          Quote quote;
          Trade trade;
        };
      };

      #pragma pack(pop)

Now because I've disabled padding in the structs, the types will map directly to the proper byte locations as specified by the data vendor. There is no need for me to manipulate bytes!

And I can get away with a single cast:

      Data* data = reinterpret_cast<Data*>(raw_buffer);
      switch (data->data_type) {
        case Data::DataType::kTrade: {
          Trade& trade = data->trade;
          std::cout << trade.symbol << ' ' << trade.price << ' '
                    << trade.size   << std::endl;
          break;
        }

        case Data::DataType::kQuote: {
          Quote& quote = data->quote;
          std::cout << quote.symbol << ' ' << quote.bid << ' '
                    << quote.ask    << std::endl;
        }

        default:
          std::cerr << "Unknown message type: "
                    << static_cast<char>(data->datatype) << std::endl;
      }

So now the code resembles a match statement in OCaml, but will actually compile directly onto the underlying bytes. No data marshaling or anything like that.

<comment><USER3> This is incredibly common in embedded programming. Lots of times you'll have some binary packet come over the wire, the first field is a packet type specifier, you can then just cast the rest of your byte buffer to the proper type.

It is _very_ unfortunate that C (and until recently C++) did not have a way to specify the size of enums. Basically this has resulted in crappy #defines being used for far too many years.

I always called what you did above "tagged types". A number of type systems actually work like that under the covers, objects get a type ID, and then a lookup function figures out what that type ID maps to.

<comment><USER4> The more common term is "tagged union". [http://en.wikipedia.org/wiki/Tagged_union](http://en.wikipedia.org/wiki/Tagged_union)

</comment></comment></comment><comment><USER5> I prefer to have a single class which stores an SI unit, as in

      final class Temperature {
        private final double kelvin;

        public static Temperature fromKelvin() { ... }
        public static Temperature fromCelcius() { ... }
        public static Temperature fromFahrenheit() { ... }

        public double getKelvin() { ... }
        public double getCelcius() { ... }
        public double getFahrenheit() { ... }
      }

This avoids the quadratic explosion of conversion methods.

A more complete example - [https://github.com/dlubarov/daniel/blob/master/data/src/dani...](https://github.com/dlubarov/daniel/blob/master/data/src/daniel/data/unit/Duration.java)

<comment><USER6> But this solution doesn't solve the original problem described in the article. With your API, it is easy for a programmer to use a temperature in Celcius as a temperature in Kelvins and the type system can't catch it.

<comment><USER5> Well, most code can just pass around Temperature objects without dealing with any particular units. There's no opportunity to mix up units there.

When some code needs to actually get a number out, say for the purpose of logging, it's possible to screw up with either approach.

With the author's classes, you could do

      void printTemperature(DegCelsius temperature) {
        log("Temperature in Kelvin: %f", temperature.getDegrees());
      }

With this Temperate class, you could do

      void printTemperature(Temperature temperature) {
        log("Temperature in Kelvin: %f", temperature.getCelcius());
      }

As the author says, "wrong code should look wrong." I think both of these look pretty wrong.

<comment><USER6> With the author's code you can define a print() function inside each temperature class, so that there is no mixup of units during printing. Or let the class supply its suffix instead of hardcoding it in an external print() function. It might not be always possible to follow that technique, but in general, it is better (according to the article, and I agree) to retain as much information at compile time as possible.

</comment></comment></comment></comment><comment><USER7> The Boost.Units[1] library takes this even further, allowing you to do things like divide a length by a time quantity and get a velocity, again with compile-time type checking.

[1]: [http://www.boost.org/doc/libs/1_55_0/doc/html/boost_units.ht...](http://www.boost.org/doc/libs/1_55_0/doc/html/boost_units.html)

<comment><USER8> F# has similar also: [http://en.wikibooks.org/wiki/F_Sharp_Programming/Units_of_Me...](http://en.wikibooks.org/wiki/F_Sharp_Programming/Units_of_Measure)

</comment></comment><comment><USER9> Did anyone else recoil in horror at how a simple function is turned into a type hierarchy? This is everything that's wrong with Java to me. Classes are all fine and well, but don't take it to extremes. Maybe the examples were just too trivial, but it really is a far greater evil than just using well named variables and a simple conversion function.

<comment><USER10> It's nice how in Go you can alias a type to int, yet it is its own distinct type with compile time checks and explicit conversions. Definitely takes away all the boilerplate.

<comment><USER11> > Definitely takes away all the boilerplate.

Until you need to write generic code. Then it is casts everywhere and boilerplate to satisfy interfaces.

<comment><USER12> >... and boilerplate to satisfy the interfaces.

Go actually provides facilities for embedding types which helps remove some of the unnecessary boilerplate from satisfying an interface.[1]

Through embedding you can leverage the implementation of an existing type to satisfy an interface w/o writing methods to dispatch to that underlying type.

[1]: [http://golang.org/doc/effective_go.html#embedding](http://golang.org/doc/effective_go.html#embedding)

<comment><USER11> I used Go quite a lot until version 1.0 got released.

You are not explaining nothing new to me.

</comment></comment></comment></comment></comment><comment><USER13> To take this to the current state of the art, look at Haskell, Agda, and Idris. Latter two being dependently typed.

Steps for learning Haskell:

[https://gist.github.com/bitemyapp/8739525](https://gist.github.com/bitemyapp/8739525)

<comment><USER14> Any recommendations for web programming in Haskell?

<comment><USER15> I highly recommend taking a look at the Happstack Tutorial. There's both a Lite version [1] and a full Crash-Course [2].

[1] [http://happstack.com/page/view-page-slug/9/happstack- lite](http://happstack.com/page/view-page-slug/9/happstack-lite) [2] [http://happstack.com/docs/crashcourse/index.html](http://happstack.com/docs/crashcourse/index.html)

</comment></comment></comment><comment><USER16> This is a good trick but it needs to be used judiciously or you'll end up with lots of boilerplate.

I particularly like Go's type system, though, because it makes it very simple:

      type celsius float64

There's no overhead and it doesn't attempt to prevent you from doing any calculations as a float64; the type checking just prevents direct assignments from a celsius type to another type.

<comment><USER17> On the other hand, Go doesn't have a way to write code that abstracts over types, and my (albeit limited) experience with Go is that most of the boilerplate is due to the language's poor type system. If you use a language with a better type system (Haskell is the first that comes to mind, though there are others), you can actually do this sort of thing without the boilerplate code.

<comment><USER18> Haskell still has some boilerplate - actually quite a lot if you don't use -XGeneralizedNewtypeDeriving (which has known problems when used with other extensions). With newtype deriving, this is about as simple as it gets:

        {-# LANGUAGE GeneralizedNewtypeDeriving #-}

        class Degrees deg where
          toK :: deg -> K
          fromK :: K -> deg

        newtype K = K { unK :: Double } deriving (Show)
        instance Degrees K where
          toK = id
          fromK = id

        newtype C = C { unC :: Double } deriving (Show)
        instance Degrees C where
          toK = K . (+ 273.16) . unC
          fromK = C . (flip (-) 273.16) . unK

        newtype F = F { unF :: Double } deriving (Show)
        instance Degrees F where
          toK = K . (+ 273.16) . (* (5/9)) . (flip (-) 32) . unF
          fromK = F . (+ 32) . (* (9/5)) . (flip (-) 273.16) . unK

Without newtype deriving (or other generic deriving extensions), one would need to implement instances of {Num, Real, Fractional, RealFrac, Floating, RealFloat} for all three types, which do nothing but map to the Double equivalents.

<comment><USER15> GeneralizedNewtypeDeriving's problems are fixed in the newest GHC which just had it's first RC a few days ago. It introduces a "roles" system which tracks which types are allowed to use GND without breaking abstraction boundaries. Roles are mostly invisible to end users as well.

</comment></comment></comment></comment><comment><USER19> "'Dog is-a Mammal' ... actually fairly sound"

I disagree in the context of OOP inheritance.

(This example uses shapes, because the idea of mutating mammals gets a little strange.)

Let's say you have a Circle class and an Ellipse class. A Circle inherits from an Ellipse, of course, because a Circle is-a Ellipse. By specializing as a Circle, we get extra reader methods such as getRadius(). Great.

But what about mutators? For an Ellipse, it may have a mutator called squash() that holds the major axis constant and halves the minor axis. But that leaves us with a problem, because circles can't be squashed while remaining circles.

Let's say we have a program like:

        void f1() {
          Circle myCircle(10.0);
          f2(myCircle);
          cout << myCircle.getRadius() << "\n";
        }
        void f2(Ellipse &e) {
          if (...)
            e.squash();
        }

Inheritance says that f2 must also accept a Circle. But what happens when we try to squash it? We can either throw a runtime error there, or we could let the squash succeed and the getRadius() in f1() will fail.

Although a Circle _value_ is an Ellipse _value_ , a Circle _variable_ is not an Ellipse _variable_. In fact, inheritance for variables should go in the opposite direction as inheritance for values, because an Ellipse variable can certainly hold a Circle value.

<comment><USER20> The circle vs ellipse problem is rather artificial and, if anything, it shows that modeling with types and everyday intuition are two different things.

If the rest of your program can handle general ellipses, it should also be able to handle ellipses having the same minor and major radius (i.e., circles). The obvious solution is to not have the Circle class and _maybe_ equip Ellipse with IsCircle() method. (Though, why would you care?!)

See the C++ faq lite items 21.6 -- 21.8. (A quote from 21.8: "Here's how to make good inheritance decisions in OO design/programming: recognize that the derived class objects must be substitutable for the base class objects. ")

<comment><USER21> The circle/ellipse problem is real and shows how mutability ruins people intuitions about models and relationships. Note that mutability ruins programs in the same way, introducing subtle inconsistencies and invariant violations.

</comment></comment></comment><comment><USER8> > The only problem is that you end up with a variable that fails to describe > itself better than "I'm a number".

Our application servers need to know about updated entities, so we broadcast notifications when an entity has been updated - but we prefer to leave it up to the individual applications to retrieve the updated entity as they see fit.

So we're often dealing with numerical ids of entities, and we've had a few bugs arise from code like so

       public void something(long accountId, long websiteId)

When people have accidentally used the wrong long in the wrong place. So we've now moved to using simple wrapper types around aforementioned longs -

        public void something(AccountId accountId, WebsiteId websiteId)

It does come at a little bit of cost as we're passing these things around a messaging system, so now we have to think about how they are serialized (Java serialization is problematic, and there are occasional religious wars about GPB vs. JSON), but it leverages the type system to make it every explicit what this number represents.

<comment><USER22> This method also create a lot of object churn. For reasonably small systems that can be okay, but you have to be mindful of it.

<comment><USER23> Would it be possible to do this as a type-alias type thing that disapears at compile time? As far as I am aware, Java has no support for such type aliasing, but it should be possible to add a pre-compiler phase to your build process that replaces these types with what they are aliases of. The only part of this that seems complicated is the type checker, which would need to be aware of the difference between AccountId and long.

<comment><USER24> I could see it being done with annotations + processing them perhaps, as some frameworks do for database properties such as index/uniqueness. An object for every scalar datatype is just overkill, even if it solves a legitimate problem.

<comment><USER22> You could make it a pre-compile step in Maven, maybe. But it'd be gross.

</comment></comment></comment></comment></comment><comment><USER25> The big problem (other than verbose code) with rolling up stuff into classes in C++ is that performance suffers. Many platforms will not pass small structs efficiently.

Inspired by Haskell's newtype, I drafted a proposal for C++14 which would have introduced native newtype to C++, but it got rolled into an omnibus paper (n3635) and appears to have been forgotten.

The proposal is modelled on strong enums, but generalizes them to all types. They are essentially strong typedefs with none of the weaknesses of wrapper types, restrict pointers, typedefs, and pragma disjoint.

        newtype NotInt : int;
        NotInt k(5); // call 'default'
        NotInt bar(NotInt);
        bar(k); // passes exactly as an ABI int would
        int baz(int&);
        baz(k); // Error! no conversion

While drafting this proposal, I realized it was a much more C++-like way of bifurcating aliases than 'restrict' keyword:

        newtype D1 : double;
        newtype D2 : double;

        void biz(D1 *x, D2 *y) {
            // Being separate, D1/D2 can be
            // assumed not to alias by optimizers
    	...
        }

While playing with it, we came up with this proposed syntax (although I don't think this would get through the committee):

        [template <typename T>]
        newtype [NotT] : [(public|private)] T [= (default|explicit|deleted)] [{
          // only non-virtual member functions, no data or reference members (something like POD)
          // this->value has all of the operations of T, but only aliases with the new type, and implicitly converts to/from T prvalues.
        }] [optional_variable_name];

<comment><USER26> I don't think there's going to be any performance difference at all compared to the basic case, which actually is another reason why this is so good. Ultimately, all these struct just hold a double, which means that in memory they're exactly a double and nothing more, there's no type tag or anything like that. And all the code is static so there is no vtable, just the double value. The compiler does the rest, not the runtime.

(Still, I'd love to see newtypes in C++)

<comment><USER27> It can make a difference due to weird ABIs. For example, in the ARM procedure call standard, 64-bit values can be returned in r0 and r1, but 64-bit structures can't (no matter whether they contain two 32-bit values or just one 64-bit value). However, I'd call that fairly niche - it's hard to imagine an application that would notice a significant performance difference.

<comment><USER3> This is very nice to know actually, and quite relevant to code I am working on right now!

Disappointing though.

> it's hard to imagine an application that would notice a significant > performance difference.

One hopes any function call made often enough to be a perf hit from this would also be designed to be easily inlined!

<comment><USER25> Scientific computing will hit this sort of problem just because the kernel is so large it actually needs to be broken into sub-procedures for icache reasons. But also, the penalty is large, because there is a full store/load cycle involved, which will clear caches and increase memory bandwidth requirements.

</comment></comment></comment></comment></comment><comment><USER28> The best write up I ever heard for this approach (often called "primitive obsession" \- [http://sourcemaking.com/refactoring/primitive- obsession](http://sourcemaking.com/refactoring/primitive-obsession)) was in Object Calisthenics, an essay by Jeff Bay in The Thoughtworks Anthology ([http://www.amazon.co.uk/The-ThoughtWorks-Anthology- Technolog...](http://www.amazon.co.uk/The-ThoughtWorks-Anthology-Technology- Programmers/dp/193435614X)):

 _" an int on its own is just a scalar, so it has no meaning. When a method takes an int as a parameter, the method name needs to do all of the work of expressing the intent. If the same method takes an Hour as a parameter, it's much easier to see what's going on. Small objects like this can make programs more maintainable, since it isn't possible to pass a Year to a method that takes an Hour parameter. With a primitive variable the compiler can't help you write semantically correct programs. With an object, even a small one, you are giving both the compiler and the programmer additional info about what the value is and why it is being used.

Small objects like Hour or Money also give us an obvious place to put behavior that would otherwise have been littered around other classes."_

</comment><comment><USER29> It might be because I only use C family languages for very low level highly optimized stuff, but I found it very weird that a blog post about types would force all those useless casts from int to double.

9 / 5 in lieu of 9. / 5\. or 1.8 will come back and bite you!

</comment><comment><USER30> I think the major problem this runs into is that it doesn't necessarily help with the actually hard parts of programming.

That is, sure, mistakes have been made with units. More mistakes are made in other areas, though. Usually amusingly more basic areas.

The question seems to be whether encoding at the type level will help with these areas. It seems the conflict is the idea that fully proving a solution is superior to partially proving it.

Laudable, to be sure, but anyone that has attempted to prove that quick sort works knows, it isn't easy. Just understanding the shorter versions of some programs takes a fair bit of brain power as it is. Understanding the fully specified version can be orders of magnitude more complicated.

So, I am left wondering if it isn't enough to stop at "passes all the tests." Especially if you have the correct tests.

And eventually one wonders whether the "correct tests" includes more stringent types. Maybe it does. Maybe it doesn't. Likely depends on the situation, is my bet.

<comment><USER31> Type checking always happens at some point in every language, dynamic or static. At some point, a routine gets executed on some form of data, and the success of that routine is predicated on the data being of the right type.

In dynamic languages, you end up writing a whole raft of tests that are not much more than type assertions that are otherwise done for you by the compiler in a statically typed language.

For statically typed languages, the type check _is_ a type of test that just so happens to be very succinct and easier to write than it is to skip. In dynamic languages, it's harder to write a test for a type assertion than it is to trust the duck is a duck, give it a punch, and hope it quacks.

It makes sense, especially for a reusable library, to try to offload as much to the static type checker as possible. It always runs and can't be skipped. As you say, "especially if you have the right tests," a static type checker forces you to have the "right tests" for at least a small part of your code, leaving you to write fewer tests for the rest of it.

<comment><USER30> I don't think you really wrote anything I did not acknowledge. Yes, types are a form of complete test over the shape/type of data that they encode. If you can encode your data with the correct type, it will go a long way to making sure you don't have errors in the types.

However, I take issue with this statement: "For statically typed languages, the type check is a type of test that just so happens to be very succinct and easier to write than it is to skip. " For the simple cases, I agree with this. But when you see the type gymnastics that some programs go through to guarantee some traits, it is hard to agree with any claim of "easier to write." Consider a "true" quicksort in haskel[1].

My assertion was that the "hard part" of programming is often in the logic, not necessarily the shape/type of the data. When you see attempts at getting the logic of a program into the type system, things start to take a turn for the worse. Really worse when you find you have many types that all encode the same type of data. (Consider the metric system debate.)

My point about "having the right tests" is that sometimes you only care/have the knowledge to think about/whatever certain scenarios that are either likely or guaranteed to happen. Pushing everything in the type system implies that you had the energy to think of and consider everything. My assertion is often you don't have that time/energy.

So, sure, you will have the "right tests" in that scenario to an extent, since you will have all tests. However, I wonder if most programs couldn't work out with a much smaller subset of those tests.

A direct analogy for the point I am making is physics. Classical/Newtonian physics break down and don't work at a level. Worse, they rely on a lot of simplifications and actually describe what is happening moreso than how/why. Yet, for a large portion of calculations that people will do, they work. I grant that encoding all of the assumptions into the calculations will make them more accurate, but often you don't need that level of accuracy. And they definitely make them harder to understand.

[1] [http://www.haskell.org/haskellwiki/Introduction/Direct_Trans...](http://www.haskell.org/haskellwiki/Introduction/Direct_Translation)

<comment><USER15> The true quicksort example you linked has very little to do with Haskell's type system and everything to do with purity. The qsort version at the bottom of that page is also pretty close to the version used in any mutable language, just with a bit more noise in the syntax. A direct translation to Python might make the syntax easier to understand, though it should be stated that this is not supposed to be pretty Python code---it's literally just a syntactic swap.

        def lscan(ary, p, h, i):
          if (i < h):
            if (p < ary[i]):
              i
            else:
              lscan(ary, p, h, i+1)
          else:
            return i

        def rscan(ary, p, l, i):
          if (l < i):
            if (ary[i] < p):
              i
            else:
              rscan(ary, p, h, i-1)
          else:
            return i

        def swap(ary, i, j):
          ary[i], ary[j] = ary[j], ary[i]

        def sloop(ary, p, l, h):
          if (l < h):
            l1 = lscan(ary, p, h,  l)
            h1 = rscan(ary, p, l1, h)
            if (l1 < h1):
              swap(ary, l1, h1)
              sloop(ary, p, l1, h1)
            else:
              return l1

        def myqsort(ary, lo, hi):
          if (lo < hi):
            piv = a[hi]
            i = sloop piv lo hi
            swap i hi
            myqsort(ary, lo, i-1)
            myqsort(ary, i+1, hi)

<comment><USER30> My point is pretty much about the "a bit more noise in the syntax." And qsort will only work on array types, something that is typically shunned by people trying to "do all of the work with the type system." Consider the types for "NonEmptyList" and such.

Is it neat when these work? Certainly. Do I think they pay obvious dividends in the effort increase? Not so much. I am open to the argument, but I have yet to see anything compelling. And with so much traction in other areas, I am doubtful that it is anything close to the silver bullet that advocates make it out to be.

<comment><USER15> Again, that noise is entirely an artifact of purity and syntactic choices, not the types.

I'm not sure exactly what you're arguing for here, but I can explain a use case of NEL—when you consume it you don't have to check the for emptiness. That means that you can eliminate an entire branch of code, either locally or in exception catching, or an entire failure mode.

Frequently people just throw those checks in and then assume, trusting their own eye or documentation to minimize the emptiness checking, but without expressing that into the type the compiler won't be able to help you not repeat that check. It's optimal to check it exactly once and people are liable to check it either 0 or many times.

<comment><USER30> My argument is that as you get more types that cover states or behavior of your program, one typically finds more code.

Is there an argument that these are beneficial? Of course there is. I just have not been privy to an example where it carried its weight.

Take the not-empty list. Typically the "cost of checking for empty" is so negligible that to push that thinking into the compiler just doesn't lead to any gains. Not even just marginal gains. It is nigh unmeasurable.

Compound this with similar types over the state of any other object, and you have an explosion of types that becomes unwieldy. Account, NotEmptyAccount, OverdrawnAccount, DrawingInterestAccount... It gets to be just crazy. Especially in areas where you don't care about any of those details, but now have to account for an inheritance tree.

And then you get into the fun of having to logic not just about how the actual program works, but how the interaction of various types works.

So, you keep saying the syntax I was referring to was due to purity. Which is kind of the point. These languages use syntax and types to help ensure purity. Which makes them more to handle for some algorithms. Less for others. And then just hard to really know the true computational cost for a lot.

Which is all to say that it is a design tradeoff. And should be undertaken as an informed decision. Not a "let the type system do the work" kind of one. More of one that "this is so important we need to make sure we can afford the cost of pushing it into the type system." Or, if you would prefer "this particular instance is so cheap to push to the type system, we should do so."

<comment><USER15> I agree completely that it's a tradeoff. I also only bring up purity to suggest that other strongly typed, impure languages could solve this with less trouble.

NEL is a particularly interesting point of compromise. I don't think it carries its weight very frequently at all, though I have used it very effectively in a few cases. I also think the burden of type hierarchies as you're describing isn't much felt in Haskell. You typically don't build large taxonomies, or taxonomies at all, in a language without subtyping.

The only place where I see taxonomies forming at all is in `lens` and there they are used to great effect.

<comment><USER30> Apologies for not seeing this. I really need to get better at following HN threads. (Or not, depending on your take.)

I think we are mostly in agreement. I just have not had the fortune to work in an area where these have panned out that well. Hopefully I get a chance to change that someday. (Though, I am currently under the song of the lisp sirens.)

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER32> Some talk on the racket mailing list was had for "numbers with dimensions[1]" in which contracts were being bent to do calculations which respected units [2]. The Frink language was also mentioned [3].

[1] [https://groups.google.com/forum/#!topic/racket- users/mnId0ux...](https://groups.google.com/forum/#!topic/racket- users/mnId0uxRaTU)

[2] [https://github.com/Metaxal/measures#4-dimensions-and- contrac...](https://github.com/Metaxal/measures#4-dimensions-and-contracts)

[3] [http://futureboy.us/frinkdocs/#SampleCalculations](http://futureboy.us/frinkdocs/#SampleCalculations)

</comment><comment><USER31> I did just this very thing in C# a few months ago[1]. The problem I had was that it gets out of hand very quickly. I wanted to be able to divide distance by time and get speed out of it, but it is extremely difficult to satisfy all of the M-to-N relationships between types and still end up being usable.

I ended up abandoning it for the project I'm building, as I wasn't too clear on what the type of a secant of an angle in degrees should be and nobody on the interwebs seemed to care enough to take notice or answer my questions.

[1] [https://github.com/capnmidnight/UnitsOfMeasure](https://github.com/capnmidnight/UnitsOfMeasure)

<comment><USER33> F# has this, I wonder how they did it?

<comment><USER34> They couldn't do it. The compiler does static verifications that, say, a value of type Meter when divided by a value of type second results in a value of type Meter/Second, where '/' is a custom concept built into the F# compiler. The .NET runtime knows nothing about these custom types.

<comment><USER31> Yeah, that's pretty much exactly the issue I ran into. The problem is that the standard .NET type semantics can only infer "up" the inheritance hierarchy.

I had gotten really close. I had types like

        Base
        Length: Base
        Meters: Length
        Feet: Length

        Compound<Base, Base>
        Area<T>: Compound<T, T> where T: Length
        SquareMeters: Area<Meters>
        SquareFeet: Area<Feet>

and had defined all of my math as generic extension methods of the Base class so that the types would compose:

        Compound<T, U> Multiply<T, U>(T a, U b) where T : Base where U : Base
        T Divide<T, U>(Compound<T, U>, U b) where T : Base where U : Base

That went a long way towards getting fairly concrete types out of only a few lines of code. But I couldn't get it the rest of the way. C# explicitly forbids user-defined typecasts between types in an inheritance chain with each other, so while multiplying two Meters got me a Compound<Meters, Meters>, it was not then possible to take a Compound<Meter, Meter> and convert it to anything like Area<Meters> or SquareMeters automatically.

<comment><USER33> Interesting problem! I was wondering why even define things like SquareMeters, but as you say the issues is reducing the terms that have different forms but mean the same thing.

<comment><USER31> Having a SquareMeters type was meant to reduce complexity. Instead of always dealing with Area<Meters, Meters> all the time, it was meant to be a shortcut.

Hrm, what about namespace aliasing? "using SquareMeters = Composite<Meters, Meters>"? Since the problem is not the composition of the type but the verbosity of the deeply composited types, then perhaps it's just about making a shortcut for the name.

<comment><USER33> things can be composed different ways. m/s^2 should equal (m/s)/s

<comment><USER31> Okay, I've definitely convinced myself there is no good way to do this without having to write way too much code over and over again. The problem is basically equivalent to inserting items into a sorted list, but you only have one operation in which to do it. So, you are either limited to simple types, or your code for operations on types explodes trying to define all of the ways to compare types (in a generic sense) and different scenarios for combinations of types.

This requires a Turing-complete type system.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER35> Might be interesting to compare to F#'s units of measure: [http://msdn.microsoft.com/en- us/library/dd233243.aspx](http://msdn.microsoft.com/en- us/library/dd233243.aspx)

</comment><comment><USER36> To take this even further, look at OCaml's strong, inferred typing.

</comment><comment><USER19> Does static typing have inherent costs?

For instance, there are some desirable properties often present in dynamic languages like hot code loading, or the ability to make code updates independently. Not many statically-typed systems are good at these things, or at least the static typing doesn't work well across boundaries (like a network, etc.).

Can these be reconciled? Can something have all of the benefits of, say, haskell and erlang at once?

<comment><USER15> There's Cloud Haskell which is an attempt to do Erlang style distributed concurrency in Haskell. For non-distributed concurrency, you can already get away with using Haskell's green threads and channels. Furthermore, there's the beginning of a hot code swapping system in GHC now, but it's not terrifically popular at the moment---it was built for a particular use at Facebook and hasn't gotten too far from there.

It's worth talking about why Cloud Haskell is tough, though. The primary issue is that Haskell doesn't have any kind of transparency into functions---if you are given a type of `(a -> b)` there's no way to examine what it is on the inside, no concept of source that could be sent to another node.

This limits the kinds of messages and spawning that can happen in a distributed Haskell system. The Cloud Haskell project is working to remove as much of this limitation as possible, though.

</comment></comment><comment><USER37> In some environments beware of creating new types and objects: [http://developer.android.com/training/articles/perf- tips.htm...](http://developer.android.com/training/articles/perf- tips.html#ObjectCreation)

<comment><USER38> This seems to paint the JIT as something that hurts performance instead of helps it, is that true on Android?

<comment><USER37> It relates to the behavior of the garbage collection scheme used by dalvik which is a mark and sweep approach. When the GC runs, that's all the VM is doing. It's pointed out that a generational garbage collector would be more efficient allocating memory presumably there is a trade off in terms of the overall memory footprint.

</comment></comment></comment><comment><USER39> Static type checking is a double-ledger bookkeeping of "what I am/have is what you are/want." Erlang and Haskell have the most interesting approaches to typing: optional and extensible respectively.

</comment><comment><USER40> I often wonder how the "tool" named MONADS could help to deal with this quite old problem of what I call

meta-types or semantic types

(i'm sure there is a proper name for this; I'm not a cs/type-theory professional ;)

Time handling f.e. and often physical Units would quite often benefit from much stronger guaranties at compile and runtime. And no, OOP and it's tools are not sufficient to handle this... in theory maybe, not in day to day development.

We need to translate the core ideas of a monad into more a "mainstream" syntax and runtime env. to prevent the next incident of:

I'll just add the integer 365 to my certificate valid-date within our core cloud infrastructure code...

I'm not necessarily a MS fan, but would love to see how Anders Hejlsberg f.e. would tackle this kind of language-design challenge ;)

<comment><USER15> I don't think this has much at all to do with monads.

<comment><USER40> Care to explain in a little more detail? Is a calender resp. date-time monad really such a far fetched idea?

<comment><USER15> That's just a state monad, the thread and what I assumed and may have assumed incorrectly you were talking about, was around overloading math operations to work on dimensionalized quantities like feet and seconds and the like. You can dimensionalize time as well—take a look at the time or thyme Haskell packages for some fascinating examples.

Doing computation in the context of the current time is absolutely monadic. If you're handling time abstractly, it's the state monad. If you're handling it for real then you need some kind of monad protecting communication with NTP/your local CPU clock. That's IO in Haskell.

<comment><USER40> Yup, you assumed incorrectly. I can see the __qualitative __difference between operator overloading and monads. That 's why I suggested to transfer the core idea of monads into more mainstream languages to provide more "semantic" safety, so to say.

Thanks for the clarification.

</comment></comment></comment></comment></comment><comment><USER41> What he's talking about is avoiding primitive obsession. Always good advice, although I find its greatest benefit comes in typing collections, rather than just using primitives like List or ArrayList.

</comment><comment><USER42> Interesting. What about more complex units, e.g. acceleration - m/s^2?

</comment><comment><USER43> Maybe it's just something I don't understand but I would go with Celsius everywhere with converting to Fahrenheit/Kelvin when showing/grabbing temperature to/from user.

<comment><USER44> The Celsius/Fahrenheit example isn't the best. Generally all examples based on units of measurement are probably flawed because sane systems can just standardise on SI and be done with it.

However, different coordinate systems that should never be mixed _are_ useful to keep separate. At work we deal with maps and geographic information, so there are always at least three coordinate systems: Screen, projection and geographic coordinates. Having a type system that catches mistakes there would be so much nicer than just always passing P2D around and trying to never mix different ones.

</comment></comment><comment><USER45> Sorry, what language are these example given in?

<comment><USER46> C++

</comment></comment><comment><USER47> Looks rather like a type-system abuse.

btw, there is Haskell for that.)

<comment><USER48> He claims it's to make wrong code look wrong. He uses the type system in order to prevent bugs, how is that abuse? Is the type system only there for performance in your opinion, or what's your angle?

</comment></comment><comment><USER49> This is fairly the premise of OOP on display. And fairly basic at that. Im not sure whats to be learned here except that in the real world, your representations will always need to be extended and subclassing is usually a bad hammer.

Interfaces are better but extend poorly through subclassing as well.

Composition is infinite but difficult to define over time when things change (usually resorts to more subclassing or more injection strategies).

I wish this article were valuable to solving the actual problem but its just a primer on OOP vs using primitives which is not particularly instructive or useful to solving real world problems.

</comment><comment><USER50> No.</comment>


Asking a Girl Out - Mathematical Analysis - <AUTHOR> http://pratikpoddarcse.blogspot.com/2010/02/asking-girl-out.html?src=hn

<comment><USER2> Long story short:

If you like a girl, don't worry if she is too pretty. Ask her out and stop obsessing.

Of course, also high up on the list: Dress appropriately, don't smell, and don't be a jerk.

</comment><comment><USER3> Nice try, left brain. But the right brain helps much more in emotional connections.</comment>


Ask HN: Any stigma associated with a .CO domain? - <AUTHOR>

I've got a cool idea for an app that I'm working on, and the perfect name isn't available as a .COM.  Somebody has it--but they aren't doing anything with it. They've also ignored my communication to them--which is fine.
So, the "new" .CO domain is open for this domain. Think that's ok? I haven't seen too many of them yet but maybe they'll get going soon enough. I know of http://duck.co and that's about it so far.
Any thoughts?  Thanks! <comment><USER2> For some reason, a '.com' domain nearly always looks more reputable than an alternative. Some people even expect it. Furthermore, '.co' might be confusing. It looks like it might be typo. I don't know what domain you want, but it might be worth your time thinking of an alternative. Otherwise, '.net' is better than '.co' in my opinion.

<comment><USER3> Two more options:

1) come up with a clever (must be easily memorable) variant which is available as .com. Be careful how you search as I've read about situations where the domain gets parked upon an inquiry. I just use whois & dig from the command line.

2) perhaps .biz or .info would work.

</comment></comment><comment><USER4> We just registered a .CO as the main address for our new startup. We think of it somewhat as a placeholder, as we hope to one day purchase the .COM since it's not actively in use by its current owner (but we haven't tried to contact yet). When we started, we had always called the product Rugby, but since the .COM is owned by Polo Ralph Lauren, Rugby.com would never be an option.

I think that had the .COM for our startup been taken by a pre-existing company with deep pockets and a strong brand, we would have simply chosen a different name. Incidentally, Rugby.co was already registered, and by a COLOMBIAN of all people. :)

We've all heard of sites that started with less-than-ideal domain names and upgraded when they had the money, and we rarely hear any of them complain about how much they spent.

<comment><USER5> and rugby.co is just a parked domain.

Shameful for us Colombians U_U

</comment></comment><comment><USER6> Personal experience has been that it's tough for me to type .co without automatically typing .com or .co.uk

I'm sure that will change over time, but it might be something to think about. It's not like the other off-brand TLDs - they tend to look nothing like .com.

</comment><comment><USER7> It makes me think the website is Columbian, and looks like a typo, not .com, .co.uk but .co. Not sure how the average person would cope! Nice for making domain hacks though like bron.co or fias.co.

</comment><comment><USER8> I'd steer well clear; even experienced internet users will remember .com rather than the actual address.

...and of course drive up the market value of the .com through type-in traffic as well. Delicious.com's owner made a pretty profit out of the risky del.icio.us domain hack, and at least people familiar with the site noticed the dots were there...

</comment><comment><USER5> .co domains can't have 'Whois Privacy Protection'.

This can be a deal breaker for some people.</comment>


Ask HN: Share your thoughts on this startup idea  - <AUTHOR>

An editorial blog for startups (i.e. TechCrunch) specifically focused on early, early stage startups (i.e. YC companies). <comment><USER2> just do it and see if you get any traffic

</comment><comment><USER3> Just start writing. See where it takes you.

</comment><comment><USER4> Sounds like a good feedback mechanism. I am working on a startup and could use this. good luck.</comment>


Iron and the Soul - <AUTHOR> https://www.oldtimestrongman.com/articles/the-iron-by-henry-rollins/ <comment><AUTHOR> Not tech centric but words that strongly resonated with me while deep in the startup trenches.</comment>


Observable - <AUTHOR> https://observablehq.com/ <comment><USER2> New venture from @mbostock, @tmcw and @jashkenas. Probably based on d3.express, which Bostock announced earlier this year but is still in private beta: [https://medium.com/@mbostock/a-better-way-to- code-2b1d2876a3...](https://medium.com/@mbostock/a-better-way-to- code-2b1d2876a3a0)

Looking forward to it!</comment>


Dead Man Driving - <AUTHOR> http://www.menshealth.com/men/best-life/life-lessons/highway-traffic-safety/article/ebe07febcb6c4210VgnVCM10000030281eac?print=true&url=http://www.menshealth.com/men/best-life/life-lessons/highway-traffic-safety/article/ebe07febcb6c4210VgnVCM10000030281eac

<comment><USER2> The article mentions that drivers usually fail to apply the brakes fully when attempting a panic stop. Some cars include brake assist systems that automatically apply full brakes when the driver presses the pedal suddenly enough.

I had the opportunity to give such a system a real-world test a couple years ago. I came over a crest on a German autobahn at just under 100 MPH (160 km/h) in a Volkswagen Crafter (a large van) to find a traffic jam in front of me. It's an odd experience; there's a snapping noise, and the pedal pressure is reduced to almost nothing. For an instant, I actually thought something had burst in the hydraulic system and the brakes had failed. The system applied the brakes as hard as the ABS would let it until I released the pedal.

<comment><USER3> I hope that you were running that van empty.

It's completely crazy how fast people will drive vans on the German autobahn, 160 to 180 is no exception. Big problem when you're hit by a gust of sidewind. (that's when being loaded comes in handy again, but you can't really have a good stopping distance _and_ good handling at speed).

</comment></comment><comment><USER4> Summmary:

\- a million irrelevant paragraphs to make this guy seem like someone you can relate to

\- guy driving on wet road

\- picks up phone

\- looks up and sees he's about to crash into another car

\- swerves across median into oncoming traffic

\- head-on collision with a Chevy Silverado

\- more irrelevant stuff about what goes on in those microseconds of the crash to try to scare you

\- explanations of what you should do in inevitable crash events

Editor anyone? Interesting read anyway, but I probably read every 3rd or 4th sentence.

<comment><USER5> I beg to differ. The article did a good job of blending physiological fact, psychological data and accident statistics in a coherent narrative.

As for the stuff making you relate to the guy - well, the whole point is that everybody believes it can't happen to them. The idea is - every fatal crash has a backstory, and yours doesn't protect you from it.

[Edit: the last link in that article might be of more interest to HN too: <http://www.zoomsafer.com>, which on quick perusal seems to be a service that automatically detects you are driving (presumably my monitoring your speed) and automatically autoresponds to calls by telling them where you are]

<comment><USER6> Hi, this is Eleanor Jones, marketing manager for ZoomSafer. Frossie, thanks for highlighting how technology like ZoomSafer's safe driving software can prevent tragic crashes by reducing mobile phone use while driving.

ZoomSafer prevents distracted driving by using your phone's GPS to detect when you're driving and automatically activating 'safe drive mode'. In 'safe drive mode', ZoomSafer locks your phone's keypad and screen so you can't text, email, or browse the web while driving - completely eliminating the need to look at your phone when your eyes should be on the road scanning for potential hazards.

The unfortunate situation detailed in the Men's Health article is one that's entirely too common - there were 6,000 deaths from distracted driving in 2008 alone. Visit www.zoomsafer.com to learn more about you can stay safely focused on the road.

</comment></comment></comment><comment><USER7> It's been many years since I took driver's ed but I don't really recall a whole lot of time spent on the subject of emergency maneuvering. It would probably be a really good idea if you have a kid learning to drive, or for yourself, to take a few lessons that include recovering from a skid (actually doing it, not just reading about it), drifting off the road, etc. etc.

<comment><USER8> I can't remember where or when I heard it but I do recall that the number of accidents increase when the driver believes that they are good enough to avoid a crash through emergency manoeuvring. Safe driving is being aware and driving in such a was so as to prevent getting into that situation in the first place and if someone is trained to recover from a skid/drifting etc... then they're more likely to take risks.

That being said, if you can handle a car in emergency situations is a good thing as long as that doesn't encourage you to cause those situations.

</comment></comment><comment><USER9> > The more accustomed you are to the road, the more difficult it is for you to > imagine disaster. And so it is with LaBar. His eyes stop scanning, his > attention withdraws, his response time drops from around 450 milliseconds to > about 1,300.

I noticed this in my own driving a year or two ago and I've since trained myself to become more alert as I get to intersections near my home. It's really amazing how alert I get. Unfortunately this training makes it difficult to fall asleep quickly after driving home. :)

>You're About to Rear-End the Car in Front of You

>The instinct: Brake hard

>The right response: Brake even harder

This is very dangerous if your brakes aren't in good condition. I did this last week with the SUV I normally drive. I applied the brake at full force to stop at a light.* I slammed on my brakes and on a flat, dry road surface my vehicle skipped 1/4 of the way into the intersect. I should have stopped before it.

Now I had known my brakes were acting a little weird but my dad replaced them today and I found out about 50% of the rotor surface was gone in large patches. Had I known just how bad my brakes were (I don't know anything about cars) I would have had them fixed sooner.

* I was driving the speed limit, not five over. I wasn't distracted. I did see the light change to yellow and started braking. Then the light quickly turned red. The light was way too fast for the intersection. (It's an odd intersect, there are two lights in a row and the traffic is usually heavy enough that you can't get up to 35mph to the second light during that time of day but I had a clear lane through the first intersection)

<comment><USER10> Brake full force anyway. There is no way that braking full force will cause it to take longer to slow down unless your tires skid.

What does "skipped 1/4 of the way" mean?

What does "50% of the rotor surface was gone in large patches" mean? Have you ever seen a brake rotor? Are you implying you had big holes in your rotor? The rotor does not have a special surface, it's just bare metal. The only way the surface could be gone is if you have a hole in it.

<comment><USER11> By skipped 1/4 of the way, I take it to mean he skidded 1/4 of the way across the intersection (or, alternatively, 1/4 of his car was in the intersection).

If your pads are worn through you can wear grooves into your rotor surface. Grooves in your rotors reduce braking friction and are very bad.

<comment><USER10> Grooves are very bad. But they don't reduce friction. It's happened to me, they make the most horrible grinding noise, but they stop the car quite well. Too well.

<comment><USER9> There wasn't any grooves. My brake pads are still good. Only the rotors were bad.</comment></comment></comment></comment></comment>


Supply Chain Security Is the Whole Enchilada, but Who's Willing to Pay for It? - <AUTHOR> https://krebsonsecurity.com/2018/10/supply-chain-security-is-the-whole-enchilada-but-whos-willing-to-pay-for-it/ <comment><USER2> I'm not a protectionist, but we need to start looking USA-based semiconductor supply capacity as a national security imperative. We should never lose our ability to manufacture these critical components, even it means policies that might be viewed as "protectionist".

<comment><USER3> How would that work in practice? Granted, the US might be able to manufacture critical components itself. What abou Italy? Honduras? Liechtenstein? Nigeria? Should they all produce their own hardware for national security concerns? And why stop there? Wouldn't your suggestion in essence lead to countries fully self-supplying everything from food to cars to communication equipment to power stations and so on and so on. And not just manufacture every single component, but also source every base materials, crops, fertilizer and more. Arguably food safety or energy supply safety might even be more critical than semiconductor safety. Even when you reach at level of self-supply capability, you still need a chain of trust to ensure that you're not compromised by your own government and intelligence agencies.

IMHO, such an concept would lead to a perfect protectionist regime while not giving any real enhanced security at all.

<comment><USER4> >What abou Italy? Honduras? Liechtenstein? Nigeria? Should they all produce their own hardware for national security concerns? And why stop there?

wrt computers, why not if they're capable and willing? it has increasingly obvious and critical security considerations. russia already has a fledgling domestic processor industry for these reasons:

[https://thenextweb.com/insider/2017/05/25/russia- showcases-f...](https://thenextweb.com/insider/2017/05/25/russia-showcases- first-computers-based-indigenous-elbrus-8s-processor/)

<comment><USER5> Russia's processor industry is also still dependent on foreign fabs like TSMC. [https://en.wikipedia.org/wiki/Elbrus-8S](https://en.wikipedia.org/wiki/Elbrus-8S)

It seems like the main fabricators based in Russia are still working on getting below 90nm.

<comment><USER6> Sure, not many nations would be producing current-gen chips anytime soon. But in case the fan is hit, I imagine ~15 year old tech will make do just fine.

</comment></comment></comment></comment></comment><comment><USER7> The DOD has been paying it through the Trusted Foundry Program. Probably not enough use of that, though. ;)

[https://www.dmea.osd.mil/trustedic.html](https://www.dmea.osd.mil/trustedic.html)

<comment><USER8> If you mean among themselves fine - but among others along the peripheral I think it is for a good reason. It would be something of them isn't something likely to be piggybacked on for several reasons - even aside from the increased cost from overspeccing and reduced economies of scale the DOD easily have the muscle to extort built in spyware to the specification to be trusted to if they want big contracts.

And given history believing such is not paranoid and wise in a sense of 'not being stupid' as opposed to prescience given the past history of exposed similar moves and a lack of transparency. Probably a borderline 'blasphemous' statement like other uncomfortable truths about things not working.

<comment><USER7> I had a recommendation for this during Snowden leaks: use hardware from threat model's most capable opponents. As in, assume it's all backdoored by at least one party. Most threatening are specific, nation states. Use their opponents' hardware.

If stopping Chinese hackers, use stuff from Trusted Foundry running most secure software you can.

If stopping Five Eyes and Israel, use Russian or Chinese hardware running most secure stuff _they_ support.

If not trusting anyone, use a computer made before 1997 that's not on a network and usually hidden in a tamper-evident compartment.

If paranoid, get rid of all electronic devices except those that detect electronic devices and emissions. Keep it far away from you itself sealed so it's not a point of attack. Periodically get it out to conduct a sweep. Your brain, pencil, paper, and hiding places are what you trust in this model. Even mechanical typewriters have acoustic and active RF weaknesses.

So, now you know how to be paranoid. Have at it. ;)

<comment><USER9> Why 1997?

<comment><USER7> As an opponent of NSA, I was studying them closely to assess capabilities. Anderson, Schell, and Karger were pushing "COMPUSEC" in the 1970's on the military who mostly didnt believe in it. They and NSA thought all you needed was access controls on computers plus COMSEC between them. The BLACKER VPN paper mentioned the rivalry between COMSEC vs COMPUSEC people, too. That was somewhere between 1980-1990. It wasn't until around 1990 after many pentests by INFOSEC pioneers and hackers in the media convinced them to hire their own hackers. They were still focused on crypto and software flaws since they were low-hanging fruit. I figured subversion would be high-profile targets only.

How Patriot act got passed was straight outta Nazi playbook. If a power grab, I knew secret backdoors, surveillance, and disappearances would follow. They'd launch USAP's forcing people to backdoor stuff or be held indefinitely under Patriot Act. Despite Haydens work, NSA still moves slow. I figured a few years before large-scale capabilities came online. Said 2004. Far as I recall, nothing in Snowden leaks refuted that estimate.

What about TAO? Hayden was forcing management to listen to engineers. They'd propose backdoors. Still software focused mostly, though. I estimated 1999. Later article on TAO history said 1997 was their start. Damn. Now I say 1995-1997.

Only thing left off was use obscure, less-popular hardware IP holders and terroristd probably werent using. Macs, Amigas, SGI... or just off-brand x86.

</comment></comment></comment></comment></comment><comment><USER10> "If it's a core business function — do it yourself, no matter what."

— Joel Spolsky, talking about Not-Invented-Here programming

[https://www.joelonsoftware.com/2001/10/14/in-defense-of- not-...](https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-invented- here-syndrome/)

<comment><USER11> How does this apply to the types of programming like encryption where it is oft advised to not roll-your-own?

<comment><USER12> If encryption is a core business function, you need as a prerequisite to having a business at all a couple of folks who know encryption well enough to roll their own.

</comment></comment></comment><comment><USER13> Can someone explain why a semiconductor production facility would be "punishingly" expensive to operate in the US?

Maybe the cost of electricity, but it seems like the labor cost wouldn't make a huge difference since most of the production work (I presume) is automated.

<comment><USER14> I don't see how, either. TI, Intel, GloFo, and a few others have facilities here. Sure China can do it cheaper, but IMO the race to the bottom is over. People want quality, which explains why Apple and Samsung outsell, say, BLU.

</comment></comment><comment><USER15> Once upon a time computers came with a full schematic diagram and enthusiasts could actually verify what they received. My point isn't so much the diagram as it is the culture of [showing/]knowing what's inside your black box.

</comment><comment><USER16> This seems like as good an excuse as any to post a link to the seminal "Reflections on Trusting Trust":

[https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p7...](https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf)

It's an absolute classic that I'm sure 97% of HN readers are familiar with, but for the 3% that aren't it's a really great meditation on just how difficult it is to trust any computing system, without somehow recursively assessing the trustworthiness of everything that has ever happened before.

<comment><USER7> That was a paper reflecting on Paul Kargers compiler-compiler subversion he did in the 1970's evaluation of MULTICS.

[https://www.acsac.org/2002/papers/classic- multics.pdf](https://www.acsac.org/2002/papers/classic-multics.pdf)

The seminal paper on subversion in the lifecycle was from another high- assurace, security researcher named Philip Myers in 1980:

[https://csrc.nist.gov/csrc/media/publications/conference- pap...](https://csrc.nist.gov/csrc/media/publications/conference- paper/1998/10/08/proceedings-of-the-21st-nissc-1998/documents/early-cs- papers/myer80.pdf)

The recommendations were carefully specifying what things do, implementing them in a structured way to inspect for backdoors, using safe languages to block regular vulnerabilities, using covert-channel analysis to find leaks, verifying things down to object code or transistors, letting people build from crypto-signed source, and using couriers for the hardware from trusted facilities onshore ("trusted trucks").

People ignored most of that. Even Thompson as he was obsessed with unsafe language vs Wirth whose work let you choose per module. It's coming back to bite everyone.

</comment></comment><comment> yc-kraln This is the kind of attack that the trustless.ai guys are fighting. They've secured their supply chain down to the silicon vendor at a fab in Italy. Really cool project.

</comment><comment><USER17> Is Apple willing to pay for supply chain security? Could they afford it?

        Writing for this week's newsletter put out by the SANS Institute,
        a security training company based in Bethesda, Md., editorial board
        member William Hugh Murray has a few provocative thoughts:

        1. Abandon the password for all but trivial applications.
        2. Abandon the flat network.
        3. Move traffic monitoring from encouraged to essential.
        4. Establish and maintain end-to-end encryption for all applications.
        5. Abandon the convenient but dangerously permissive default access control rule of "read/write/execute"

</comment><comment><USER18> This is going to be a huge problem. If that backdoor exists, someone will find it and use it. Not necessarily the people who put it there.

</comment><comment><USER19> The supply chain is secure. But that's not FUD that sells. The gap between mainstream news and tech news is closing.

</comment><comment><USER20> > Indeed, noted security expert Bruce Schneier calls supply-chain security "an > insurmountably hard problem."

Centrally, yes. But supply chain verification & tracking is one thing blockchains are genuinely good at. There are actual blockchain-based products on the market for that.

Imagine a company such as Apple forcing their suppliers to authenticate each production step from raw material to shipped good on a proprietary blockchain. It's certainly doable from a technical standpoint, and Apple's suppliers are probably eager enough not to lose Apple's business to comply.

<comment><USER21> I don't understand how this solves the problem:

\- if all links in the supply-chain are _properly_ verifying/tracking components, why can't they send this verif/tracking data to their client's server?

\- if they're not making a good-faith effort, what does blockchain solve?

Blockchain is genuinely good at maintaining an immutable, trust-free ledger. It's _NOT_ good at making sure the data that's written to it is true.

<comment><USER22> It does not, that's why a blockchain on it's own does not solve the problem but in conjunction with other technologies it could. E.g. food - you have to make sure your sensors (Temperature etc) are tamper proof, but you also have to find a way to make it hard to seperate the sensor from the goods. Only then it would make sense to use it as an oracle. Also if you make it hard enough to cheat the sensor at some point it's just easier to cool the goods and transport them accordingly.

<comment><USER21> Right, so it really doesn't solve anything. It just passes the buck further down.

Blockchain is a solution in search of a problem, here.</comment></comment></comment></comment>


King Size Linodes Now Available - <AUTHOR> http://blog.linode.com/2009/07/22/king-size-linodes-now-available/

<comment><USER2> Hmmm, these prices aren't that competitive with dedicated servers.

$479 will get you a dual-quad core dedicated with 16GB of ram these days. Even the lower end ones (2880MB) you can rent a 4GB SoftLayer box for the same price.

<comment><USER3> FDC Servers (fdcservers.net) is another good option for high bandwidth sites. 100 meg unmetered line (you can push 15 - 20 TB/mo) - good colo and dedicated server pricing.

</comment></comment><comment><USER4> I get AWS, AWS is scale on demand + multiple datacenters. I get the lower linode pricing levels, they're a good deal.

The large ones make no sense to me. If you're paying monthly for a bigass machine then linode is way overpriced vs. dedicated. If you need to scale up really quick then linode won't work because they bill monthly. They have redundant DCs but if one got knocked out I imagine there'd be a scramble to use all the available ones if a DC went down.

<comment><USER5> My understanding was that if you only needed a vps for five days then you're only charged for five days. Their prices are given as monthly but they measure your usage daily.

<comment><USER4> Maybe I'm missing something, but the linode FAQ says:

How am I billed?

We currently offer monthly, annual, and 24-month billing cycles. If you are interested in pre-paying, open a support ticket and we can accommodate you. When you sign up, your first invoice is pro-rated until the end of the month. After that, your subscription is automatically renewed on the first of each month (for annual accounts, on the anniversary of your first full month).

<comment><USER5> Taking another look at the linode site I can definitely see where the confusion might come in as it's difficult to find any indication that costs are calculated daily. However I did find the following, again from the FAQ:

Can I upgrade my Linode?

[...]

Removing resources pro-rates the balance of the month back to your account so you only pay for the time you've used them.

Edit: I've just gone into my linode account and the removal link does give you a figure for how much you'll get back. You pay for a month in advance but if you don't need the full month you can remove the linode and get your money back for the time remaining.

</comment></comment></comment></comment><comment><USER6> Now you can run more than one Rails app on your server!

<comment><USER7> I know this is flippant, but if you're quite keen to run more than one Ruby web app, why not try ActiveRecord (so you get the Rails ORM) with Sinatra? It's a pretty lightweight solution.

<comment><USER6> Honestly, I was trolling and expected to be modded down to -8 pretty quickly. It's funny how many of my dead-serious posts get modded down and how many of my trolls get modded up.

It's somewhat sad, actually.

<comment><USER8> If that's the case, you should know better than to post it.

<comment><USER6> It's apparently what people want to read.

The key to being a successful author is to write stuff that people want to read.

<comment><USER8> I love the clever quips over at Reddit as much as the next guy, but I've come to enjoy HN's higher level of expectation out of its responses. Just because people are modding you up, doesn't mean you shouldn't try to uphold that standard.

<comment><USER9> Agreed. If I want to read trolls and mindless comments I'll go to Reddit/Digg.

<comment><USER10> Ironically this discussion is uninteresting and a waste of space.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> What keeps me coming back to Linode is the administration interface / console.

The Plesk, cPanel and Virtuozzo consoles provided to customers by other VPS hosting companies are horrible in comparison.

Slicehost's RubyOnRails based interface is too simple and doesn't provide anywhere near the functionality that Linode provides.

<comment><USER12> For low end stuff, linode is great.

</comment></comment><comment><USER13> Awesome additions, shame the bandwidth doesn't go up proportionally but tops out at 2GB, but we can't always have everything :)

Linode still rule.

edit: Oops, meant 2TB

<comment><USER14> Of course it's 2TB (2000GB).

</comment></comment><comment><USER15> I use Linode for some of my lower end services

But what happens when your application requirements don't scale as a power of 2 ??

I need a low powered box but with 1TB of storage. 1TB storage is not even a possibility with any of the King Sizes. Amazon S3 at 1TB would be to slow, as there are many many documents all being randomly fetched.

I guess I have to go with Colo/Softlayer unless HN friends have some better suggestions??

</comment><comment><USER16> The blog post or signup form should really mention that the trailing number is the amount of RAM, in MiB, for the node, and that larger nodes get proportionately more CPU as well (assuming that's true).

Unless you back up to the home page, the charts on the blog post and signup form make it look like the only advantage from (say) 5760 to 14400 is plus-384GB of HD space for $480/month. You can't even click from the node- selector on the signup form to mode details about the nodes.</comment>


New metal hydride storage system design brings hydrogen cars closer to reality - <AUTHOR> http://news.uns.purdue.edu/x/2009a/090402MudawarHydrogen.html <comment><USER2> This post offers an interesting viewpoint about how far hydrogen fuel cells can go:

<http://blog.cocagne.com/hydrogen-economies.htm></comment>


Creating 3D models using a webcam - <AUTHOR> http://mi.eng.cam.ac.uk/~qp202/my_papers/BMVC09/

<comment><AUTHOR> A direct link to the video illustrating the process <http://www.youtube.com/watch?v=vEOmzjImsVc>

</comment><comment><USER2> This was posted here yesterday.

<comment><AUTHOR> Oops, sorry :) I tried to search, but didn't find it.</comment></comment>


Pi Day 2015: a sweet treat for maths fans - <AUTHOR> http://www.theguardian.com/science/2015/mar/13/pi-day-celebration-maths-fans-language-memory-contests

<comment><AUTHOR> Set your alarm for 3/14/15, 9.26:53 am :-)</comment>


One way to make containers network: BGP - <AUTHOR> http://jvns.ca/blog/2016/07/16/calico/ <comment><USER2> It's really not that difficult to network containers. We're using flannel [1] on CoreOS. We're using flannel's VXLAN backend to encapsulate container traffic. We're Kubernetes users so every kube pod [2] gets it's own subnet and flannel handles the routing between those subnets, across all CoreOS servers in the cluster.

I was skeptical when we first deployed it but we've found it to be dependable and fast. We're running it in production on six CoreOS servers and 400-500 containers.

We did evaluate Project Calico initially but discovered some performance tests that tipped the scales in favor of flannel. [3] I don't know if Calico has improved since then, however. This was about a year ago.

[1] [https://github.com/coreos/flannel](https://github.com/coreos/flannel)

[2] A Kubernetes pod is one or more related containers running on a single server

[3] [http://www.slideshare.net/ArjanSchaaf/docker-network- perform...](http://www.slideshare.net/ArjanSchaaf/docker-network-performance- in-the-public-cloud)

<comment><USER3> Is flannel used in Kubernetes for networking by default? Or is it something that needs to enabled and configured separately?

<comment><USER4> Kubernetes has a requirement that containers (more accurately "pods") can connect via a "flat networking space". How this is achieved varies between deployments, flannel, calico and weave are all common approaches. Kelsey Hightower's "Kubernetes the Hard Way" simply configured it at the router level: [https://github.com/kelseyhightower/kubernetes-the-hard- way/b...](https://github.com/kelseyhightower/kubernetes-the-hard- way/blob/master/docs/07-network.md)

<comment><USER3> That makes sense. Thanks for the link as well, i've been looking for something exactly like it. Looks like a great resource!

</comment></comment></comment></comment><comment><USER5> Another solution to this problem is Romana [1] (I am part of this effort). It avoids overlays as well as BGP because it aggregate routes. It uses its own IP address management (IPAM) to maintain the route hierarchy.

The nice thing about this is that nothing has to happen for a new pod to be reachable. No /32 route distribution or BGP (or etcd) convergence, no VXLAN ID (VNID) distribution for the overlay. At some scale, route and/or VNID distribution is going to limit the speed at which new pods can be launched.

One other thing not mentioned in the blog post or in any of these comments is network policy and isolation. Kubernetes v1.3 includes the new network APIs that let you isolate namespaces. This can only be achieved with a back end network solution like Romana or Calico (some others as well).

[1] romana.io

</comment><comment><USER6> On the topic of "why do we need a distributed KV store for an overlay network?" from the blog: there's a good blog post about why Kubernetes doesn't use Docker's libnetwork.

[http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt- use-...](http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt-use- libnetwork.html)

<comment><USER7> Thanks! I had almost forgotten about the container networking mayhem. Would love to find out what progress has been made over the past six months since the blog post was written.

<comment><USER8> Kubernetes now uses CNI to configure interfaces in its most common "kubenet" configuration, and (obviously) when you put it into "CNI" mode.

Most container network offerings - Calico, Flannel, Weave, etc. ship with a CNI plugin.

Docker have not altered their network plugin API.

(I work on Weave Net, including the plugins for both Docker and CNI)

</comment></comment></comment><comment><USER9> BGP seems a needlessly complex solution to this problem. VXLAN would, IMO, be a much better fit.

(--Network engineer who manages BGP for an ISP)

<comment><USER10> I don't know. With VX?LANs you need a whole lot of other infrastructure to manage e.g. routes, load balancing or tenant-specific ACLs. By coercing BGP into providing isolation, you can slap on routing, load balancing, ACLs "for free", i.e. managed by the same control plane.

If you just need isolation I agree with you. But I actually find the Calico solution rather elegant when looking at the whole package.

(--System administrator who manages VXLAN on a public cloud)

</comment></comment><comment><USER11> We're just about to switch to BGP internally using Calico (mentioned in another comment, I believe performance is good now), we run around 300-600 containers currently using our implementation using Consul+Serf. We'll drop a talk on it once we've made the switch if anyone is interested. We're deliberately avoiding flannel because of the tunnelled networking and added complexity that we don't feel we want to introduce.

</comment><comment><USER12> I've for a long time wondered if anyone has successfully just gone full ipv6 only with a substantial container/vm roll-out. On paper it should have:

1) enough addresses. Just enough. For everything. For everyone. Google-scale enough.

2) Good out-of-the box dynamic assignment of addresses.

And finally, optional integration with ipsec, which I get might in the end be over-engineered, and under-used -- but wouldn't it be nice if you could just trust the network (you'd still have to bootstrap trust somehow, probably running your own x509 CA -- but how nice to be able to flip open any book on networking from the 80s and just replace the ipv4 addressing with ipv6 and just go ahead and use plain rsh and /etc/allow.hosts as your entire infrastructure for actually secure intra-cluster networking -- even across data-centres and what not. [ed: and secure nfsv3! wo-hoo]).

But anyway, have anyone actually done this? Does it work (for a meaningfully large value of work)?

<comment><USER13> You don't really need IPv6 to do this. IPv4 is sufficient; you just need to assign more than 1 IP address to your physical interfaces. This has been possible since at least the early 1990s.

The problem is that many cloud providers (ahem EC2) don't make this trivially easy like they should.

<comment><USER14> They don't make it trivial because we're out of IPv4 address space. Which would be the reason for doing this with IPv6. The automatic addressing of IPv4 is also nowhere near as simple as v6, not even close.

<comment><USER13> Both AWS and GCE can allocate private 10.0.0.0/8 subnets for internal networks (e.g. VPCs). There is no address scarcity in such subnets.

<comment><USER12> There's a big difference between routable (on the Internet) addresses and non- routable addresses. For one, you can trivially merge two different sets of resources (say from two different organizations or projects) without address conflicts, if they both have globally unique, routable addresses.

LAN networking is great. Internet networking is so much better, it has effectively given birth to a new technological age.

<comment><USER13> I don't disagree on these particular points. My contention is solely that lack of IPv6 is not a showstopper.

<comment><USER14> It absolutely is a showstopper if you don't have applications that are NAT friendly. And in 2016 if you are forcing people to use NAT and claiming that's a fix, you're doing the entire world a disservice.

<comment><USER13> I've been hearing this argument for 15 years now and it hasn't changed. Sure, there's a public address shortage these days, but the incessant whining by end-to-end purists who think NAT is an unbearable sin falls on deaf ears just as much now as it ever did.

I'm not against V6 by any means, but internal IPv4 private subnets are simply not a problem for the 99+% of who still somehow manage to get things done in the real world.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> BGP looks really complex. Isn't OSPF (BGP's "little brother") a much attractive choice here? It's still complex, but should be much simpler.

Another attractive alternative to Flannel is Weave [1], run in the simpler non-overlay mode. In this mode, it won't start a SDN, but will simply act as a bridge/route maintainer, similar to Flannel.

[1] [https://www.weave.works/products/weave- net/](https://www.weave.works/products/weave-net/)

<comment><USER16> BGP IMHO is much simpler than OSPF. No different area types, support for communities, no need to keep a link-state database for the entire network in all nodes, ...

<comment><USER17> People keep talking about link state database overhead, but how significant is this in reality? The graphs we're talking about, even in huge deployments, are small.

If you're running etcd or consul, I'm not sure you retain the right to call LSA flooding "complicated". It's simple compared to RAFT!

<comment><USER16> Size probably doesn't matter that much until you start to fill entire datacenters, OSPF nowadays should work with hundreds of routers as well. Would be interesting to see how failure cases compare, if I remember right one of the arguments for BGP in data-center fabrics was that the updates following them stay more localized. (EDIT: a description how Microsoft uses it for really large networks, slide 11 talks about surges: [http://www.janog.gr.jp/meeting/janog33/doc/janog33-bgp- nkpos...](http://www.janog.gr.jp/meeting/janog33/doc/janog33-bgp- nkposong-1.pdf))

I find BGP easier to understand, and I don't see what benefit OSPF would have. (Not that I really have non-trivial experience with either, have only used them at home and toy networks)

</comment></comment></comment></comment><comment><USER18> Have I misunderstood something here? We don't BGP on a local networks. Via ARP, a node says "who has $IP?" Something answers with a MAC address. The packet for $IP is wrapped in an Ethernet frame for that MAC address. If the IP isn't local to your network, your router answers with its own MAC, and the packet is framed up for the router.

BGP is the process by which ranges of IPs are claimed by routers. Is Calico really used by docker containers in this way?

<comment><USER19> Yeah, have an upvote. This is totally a case of "using an industrial core drill with tungsten carbide bits when all you need is an Ikea drill and a chinese cheese-grade bit".

Sounds like this guy just found out about a cool new tool and decided to blog about it. BGP can be used on local networks but it's total overkill for a docker situation where all of your instances are likely in the same rack (often on the same machine!). Ifyou don't even have an AS from ARIN/RIPE there's no reason to even touch this (as you pointed out, it's all the protocol is designed to broadcast to the _public internet_ \-- e.g. 'hey I own this AS which has rights to this net-block, direct packets in this fashion please!' Jeez.

I have no idea what the CPU overhead of running this is but I'm sure it's not trivial, especially if the BGP daemon is tuned to retain any significant amount of the whole BGP table (RAM/swap issues galore I'd imagine). Granted the article is titled 'one way'.. which is empirically true, it's a Rube Goldberg way of going about networking.

(n.b., OSPF is what people use for "BGP" within your own intranet, even when you have tens of thousands of boxes. It's called _" border gate protocol"_ for a reason..)

(Sorry, I don't use docker so I can't actually make a constructive comment telling you what the canonical/ _right_ way of doing it is, but I can assure you, this is not it.)

<comment><USER20> (I didn't downvote you, but this might be why someone did...)

Eh, BGP on local networks is common enough that it was novel maybe a decade ago. It's perfect for running on servers to announce /32 addresses upstream to ToR switches. OSPF is actually more heavyweight & complex since you have to carry link state, do neighbor elections, etc. Ref: NANOG talk in 2003: [https://www.nanog.org/meetings/nanog29/presentations/miller....](https://www.nanog.org/meetings/nanog29/presentations/miller.pdf)

You don't even need an AS from your RIR for BGP to be useful on internal networks, just pick one (or more) of the private ASNs and roll with it.

Current best practice for internal networks (on the SP side at least) is to use OSPF to bootstrap BGP by enabling IP reachability for p2p & loopback addresses. After that customer / service routes are carried by BGP, not by OSPF. This is because BGP doesn't spam link state and has better policy knobs. You get a double win because OSPF converges faster with fewer routes and if you have redundant links your BGP sessions won't flap because the loopback addresses stay reachable. Ref: NANOG talk in 2011: [https://www.nanog.org/meetings/nanog53/presentations/Sunday/...](https://www.nanog.org/meetings/nanog53/presentations/Sunday/bgp-101-NANOG53.pdf)

CPU/RAM overhead is insignificant with a bgpd like BIRD or Quagga. They work for internet-scale routing (currently over 610,000 prefixes) with trivial load. An Atom C-series with a few GB of RAM can deal with internet BGP routing updates (CPU becomes significant for packet forwarding, not so much maintaining routing tables).

I'll take a boring 10-year-old routing setup using iBGP on cluster of route reflectors running BIRD with servers numbered out of a /24 per rack routed by a ToR l3switch with each service using BGP to announce a /32 over new and exciting L2 overlay networks any day. Troubleshooting is easier without having to work through different layers of encapsulation, dealing with MTU, trusting a novel control plane for whatever overlay network your using, etc.

<comment><USER21> > CPU/RAM overhead is insignificant with a bgpd like BIRD or Quagga. They work > for internet-scale routing (currently over 610,000 prefixes) with trivial > load. An Atom C-series with a few GB of RAM can deal with internet BGP > routing updates (CPU becomes significant for packet forwarding, not so much > maintaining routing tables).

We've got a DPDK-enabled VRouter that will run over 12Mpps on a C2758 (8 core) with a full BGP table.

</comment></comment></comment></comment><comment><USER22> More on this here: [https://medium.com/@sargun/a-critique-of-network-design- ff85...](https://medium.com/@sargun/a-critique-of-network-design- ff8543140667#.2fwstossu) \-- BGP isn't just about containers. It's about signaling. It's a mechanism for machines to influence the flow of traffic in the network.

This isn't container weirdness. This is because networks got stuck in 2008. We still don't have have IPv6 SLAAC. Many of us made the jump to layer 3 clos fabrics, but stopped after that. My belief is because AWS EC2, Google GCE, Azure Compute, and others consider this the gold standard.

IPv6 natively supports autoconfiguring multiple IPs per NIC / machine automagically*. This is usually on by default as part of the privacy extensions, so in conjunction with SLAAC, you can cycle through IPs quickly. It also makes multi-endpoint protocols relevant.

Containers and bad networking because of the lack of IP / container is a well- known problem, it's even touched on in the Borg paper, briefly: One IP address per machine complicates things. In Borg, all tasks on a machine use the single IP address of their host, and thus share the host's port space. This causes a number of difficulties: Borg must schedule ports as a resource; tasks must pre-declare how many ports they need, and be willing to be told which ones to use when they start; the Borglet must enforce port isolation; and the naming and RPC systems must handle ports as well as IP addresses.

Thanks to the advent of Linux namespaces, VMs, IPv6, and software-defined networking, Kubernetes can take a more user-friendly approach that eliminates these complications: every pod and service gets its own IP address, allowing developers to choose ports rather than requiring their software to adapt to the ones chosen by the infrastructure, and removes the infrastructure complexity of managing ports.

But, I ask, what's wrong with the Docker approach of rewriting ports? Reachability is our primary concern, and I'm unfortunately BGP hasn't become the lingua franca for most networks ("The Cloud"). I actually think ILA ([https://tools.ietf.org/html/draft-herbert- nvo3-ila-00#sectio...](https://tools.ietf.org/html/draft-herbert- nvo3-ila-00#section-4.5)) / ILNP (RFC6741) are the most interesting approaches here.

<comment><USER8> > what's wrong with the Docker approach of rewriting ports

It requires that you rewrite the software trying to talk to that port, to make it aware that you've put the new port number in a special environment variable.

<comment><USER22> Have you looked at Docker bridge mode? and Mesosphere's VIPs?

What do you think of them?

<comment><USER8> Docker bridge only works between containers on one machine; this is exactly why we wrote Weave Net two years ago, to let you network simply between containers running anywhere.

I hadn't considered using Virtual IPs to reverse out port-mapping. I guess it would work provided you have good connectivity between hosts - it would be a nightmare to try to configure a firewall where the actual ports in use jump around all the time.

Also such schemes require that you know in advance which ports each component listens on, and that there are no loops in the graph. Both of these requirements can be constraining.

</comment></comment></comment></comment><comment><USER23> Or you could NAT on the host and deploy simpler overlay networking: [https://github.com/pjperez/docker- wormhole](https://github.com/pjperez/docker-wormhole)

You can deploy this on any machine (container or not) and have it always reachable from other members of the same network, which could be e.g. servers on different providers (AWS, Azure, Digital Ocean, etc)

<comment><USER24> You should probably mention that this is a PaaS.

(and maybe also that you are affiliated with them)

<comment><USER23> Hi,

Sorry, I should have made it explicit. As it's my own repo and my profile's email address gives away I'm part of Wormhole I didn't think about making a statement on the coment; but you're right.

Thanks!

</comment></comment></comment><comment><USER17> Especially since there isn't really a policy-routing component to this, isn't BGP pretty _extremely_ complicated for the problem Calico is trying to solve?

Stipulating that you need a routing protocol here (you don't, right? You can do proxy ARP, or some more modern equivalent of proxy ARP.), there's a whole family of routing protocols optimized for this scenario, of which OSPF is the best-known.

<comment> X-Istence Calico is installing routes in the Linux kernel. Those routes are pulled out and distributed using BIRD. BIRD can do OSPF instead if you'd like.

All Calico cares about is that routes are distributed across various systems, they don't necessarily care how you do it (configure BIRD however you'd like).

BGP is surprisingly simple and easy to set up with BIRD. Setting up a route reflector with local hosts on the same L2 all peering with each other and suddenly you can route whatever IP's you want by announcing them to your peers.

Why do people think BGP is complicated?

<comment><USER25> >Why do people think BGP is complicated?

Read your own paragraph before this question. Why do I need to run another process to exchange routes and configure a mesh or a route reflector? As an admin that's just another mess of processes and communication to worry about.

Just because BGP is easy for you does not mean it's easy for most server admins and devs without heavy networking backgrounds.

<comment> X-Istence Wait what? How else should we be exchanging routes? Should we shove them into a distributed key value store and then having each of the nodes pull out the routes and installing them?

> As an admin that's just another mess of processes and communication to worry > about.

Yet we fully expect admins to understand and build HA redundant clusters for databases, or how to manage and update all the machines under their control, and a variety of other tasks.

There is nothing inherently different about running a BGP speaking daemon. It's all config.

I don't have a heavy networking background at all. I'm a software engineer that's currently working as a system architect, but even I can understand something as simple as a route distribution system.

</comment></comment></comment></comment><comment><USER26> There's a lot of misinformation in this.

>A Linux container is a process, usually with its own filesystem attached to it so that its dependencies are isolated from your normal operating system. In the Docker universe we sometimes talk like it's a virtual machine, but fundamentally, it's just a process. Like any process, it can listen on a port (like 30000) and do networking.

A container isn't a process. It's an amalgamation of cgroups and namespaces. A container can have many processes. Hell, use systemd-nspawn on a volume that contains a linux distro and your container is basically the entire userspace of a full system.

>But what do I do if I have another computer on the same network? How does that container know that 10.0.1.104 belongs to a container on my computer?

Well, BGP certainly isn't a hard requirement. Depending on how you've setup your network, if these are in the same subnet and can communicate via layer 2, you don't need any sort of routing.

>To me, this seems pretty nice. It means that you can easily interpret the packets coming in and out of your machine (and, because we love tcpdump, we want to be able to understand our network traffic). I think there are other advantages but I'm not sure what they are.

I'm not sure where the idea that calico/BGP are required to look at network traffic for containers on your machine came from. If there's network traffic on your machine, you can basically always capture it with tcpdump.

> I find reading this networking stuff pretty difficult; more difficult than > usual. For example, Docker also has a networking product they released > recently. The webpage says they're doing "overlay networking". I don't know > what that is, but it seems like you need etcd or consul or zookeeper. So the > networking thing involves a distributed key-value store? Why do I need to > have a distributed key-value store to do networking? There is probably a > talk about this that I can watch but I don't understand it yet.

I think not at all understanding one of the major players in container networking is a good indication it might not yet be time to personally write a blog about container networking. Also absent is simple bridging.

Julia generally writes fantastic blogs, and I know she doesn't claim to be an expert on this subject and includes a disclaimer about how this is likely to be more wrong than usual, but I feel like there was a lot of room for additional research to be done to produce a more accurate article. I understand the blog is mostly about what she has recently learned, and often has lots of questions unanswered... But this one has a lot of things that are answered, incorrectly :(

<comment><USER4> I've never read any of her blogs before (that I can recall) and I do agree there are some misunderstandings. But it was clearly written as as a brain dump to help other people going through the same process and it largely achieves that goal. I really don't like the concept "don't write a blog unless you're an expert" \- we would lose out on lots of valuable discussions and helpful articles, especially for beginners. For example, we wouldn't have this HN thread with useful commentary if it wasn't for the author.

I think the best approach is to constructively comment and engage to improve the article. The important thing is to do this in a positive manner so that the author feels they have done something of value and started a conversation. It is surprisingly difficult to do this, and I've certainly failed on occasion, but it is definitely worth trying.

<comment><USER26> I'm certainly not suggesting the need to be an expert to write a blog - I just personally would be more cautious with some statements if I was writing on something I wasn't knowledgable in.

</comment></comment></comment><comment><USER27> The internal OpenDNS docker system, Quadra, relies on BGP for a mix on of on- prem and off-prem hosting:

[http://www.slideshare.net/bacongobbler/docker-with- bgp](http://www.slideshare.net/bacongobbler/docker-with-bgp)

</comment><comment><USER13> The real problem is that cloud providers don't provide out-of-the-box functionality to assign more than one IP to a network interface. If they did this, there wouldn't even be an issue.

I've been requesting this feature from the EC2 team at AWS for some time about this, to no avail. You can bind multiple interfaces (ENIs) to an instance (up to 6, I think, depending on the instance size), each with a separate IP address, but not multiple IPs to a single interface.

BGP, flannel, vxlan, etc. are IMO a waste of cycles and add needless complexity to what could otherwise be a very simple architecture.

<comment><USER28> It's not a waste of cycles. You are just pushing your responsibilities to AWS, the cycles and memory for routing your additional IPs are required anyway.

<comment><USER13> Why do you think that maintaining the network should be my responsibility, as opposed to the provider's?

</comment></comment></comment><comment><USER29> Oh boy. And containers were supposed to make things _easier_.

<comment><USER30> You could just have every container get a magic IPV6 address that just works.

[https://www.zerotier.com/community/topic/67/zerotier-6plane-...](https://www.zerotier.com/community/topic/67/zerotier-6plane- ipv6-addressing)

Full disclosure: this is ours.

<comment><USER31> Or you could just give every container a real IPv6 address, no need for any magic...

<comment><USER30> Nothing against that either but some people can't do it due to hybrid deployments, providers that don't give you a /64, or providers that don't offer V6 at all.

Currently Amazon, Google, and Azure have no native IPv6 support.

Also many are allergic to the security implications. You have to be rigorous with ip6tables and making sure everything speaks SSL or another encrypted protocol using authentication in both directions. Many things do not support SSL or don't support bidirectional auth.

Personally I doubt overlay networks are going away. Most backplane software like databases, distributed caches and event servers, etc. offers literally no security because it's all built with the assumption that it will run on a secure backplane. I have personally railed against this for years but I've found that it's a total waste of breath.

<comment><USER32> Which provider won't give you a /64 (that supports IPv6 at all)?

Even my residential cable provider gives me a /60\. Anything less seems absurd.

<comment><USER30> I've seen smaller ones do /128\. Probably just cluelessness.

The bigger issue is the need for a secure backplane, which will remain until all server software authenticates all sockets in a strong way.

<comment><USER32> I agree re: the secure backplane (though I would prefer to see it done outside server software in a more comprehensive manner).

I'm surprised that the bar isn't set a bit higher inside the cloud provider infrastructure for tenant separation at the network level. I suppose it boils down to the lack of assurance at an even lower level (who trusts Xen these days?) that seems unlikely to be fixed in the short term.</comment></comment></comment></comment></comment></comment></comment>