What is contravariance and why all programmers should care about it
CSI: Python Type System, episode 1 - Daftcode Blog
[s02e01] Understanding ContravarianceThis blog post is based on a real case. It all has started with running mypy type checker on some code. I got a strange message. I did not understand the message. I disliked the fact that I did not understand it 😐 So I started to investigate a little. Effects of this investigation helped me to better understand formal features of type systems. I’m also pretty sure it will help me write a better code.This blog post is the blog post I was looking for while struggling to comprehend the error. None of the sources I found really made me think: Ah, now I get it! I needed to experiment a little on my own at first. This blog post more or less retells the path I took. The first part is about my investigation and process of wrapping my head around its results. The second part is about possible code fixes. The third part is devoted to defining and explaining the key concepts in understanding the issues discussed here: covariance, contravariance, and invariance; also, you will find there links to the sources I used writing all three parts.This post is for you if:you know Python syntaxyou know what inheritance isyou know the basics of Python type system; reading the first “season” of my typing blog post series is enough (see s01e01 and s01e02)you don’t know what “covariance”, “contravariance” and “invariance” are, and…you would like to know what all those “variances” are 😉This post might not be for you if:you know what “function is covariant in the return type, but contravariant in the argument types” means 🤓I will try to go over everything step-by-step and be as descriptive as reasonably possible. As a result, this post may be wordy for some of you. I’m writing it for myself-from-a-few-months-ago, so ¯\_(ツ)_/¯. On the other hand, it won’t be easy-peasy and it will probably require some extra focus. Re-reading some passages may be required. So flex your brain-muscles in advance! 💪All code snippets are compatible with Python 3.6+. Yet, issues discussed here apply to all Python versions and generally to all programming languages. I’m using mypy 0.630 type checker.The CaseLet’s study the following code:This is a somewhat obfuscated code I ran mypy on. The real code was about adapters and events that are passed through them, but the relationships between classes were exactly the same.The code might seem perfectly fine: animal eats (general) Food, dog eats (more specific) Meat. Yet, mypy is not happy. For line marked with # mypy error comment, it reports the following error:error: Argument 1 of "eat" incompatible with supertype "Animal".Mypy, like a real Pythonista, counts from zero, so “argument 1” means food, not self. But what’s with the error? It basically means that the type of argument food of Dog.eat() is incompatible with the type of argument food of Animal.eat(). Okay, but what does that mean? Incompatible in what way? It might seem natural that more general Food is used by more general Animal, and less general Meat is used by less general Dog.At this point, you might think that if the code looks fine and works (it’s been thoroughly tested!), should you care about the error? These “theoretical” inconsistencies aren’t really helpful in real software development, am I right? Hopefully, after reading the rest of the blog post, you will be convinced that this “theory” is really helpful and might prevent catastrophic bugs. So let’s start our investigation! 🕵️️Me at work. [source]InvestigationAn ExperimentWe know that something is definitely wrong with types of function arguments which are “incompatible” across Animal and Dog classes. Let’s see what happens when we swap types of food argument: Animal will eat Meat and Dog will eat generic Food:No error. Isn’t it strange? Now, more general Animal eats more specific Meat and less general Dog eats more general Food, and everything is fine in terms of types. We need more clues! 🔎General Assignment RulesFirstly, let’s step back a little and recap assignment rules discussed in this blog post.Assigning a_dog to an_animal — a variable of type Animal— is type-safe, since every Dog is an Animal, meaning Dog is a subtype of Animal, in short: Dog <: Animal. (<: symbol reads: “is a subtype of”.) In other words, it’s type-safe to use a_dog wherever an_animal is expected.Assigning an_animal to a_dog— a variable of type Dog— is not type-safe, since not every Animal is a Dog; Animal is not a subtype of Dog.So, in general, the basic type-safe assignment looks like this:# SubType <: SuperTypesupertype_variable: SuperTypesubtype_variable: SubTypesupertype_variable = subtype_variableIt’s type-safe to use an object of SubType wherever an object of SuperType is expected.Function Assignment RulesWe know that functions have types too. So, typing assignment rules apply to them as well. Let’s define two simple functions. The first one takes an Animal and returns None. The second one takes a Dog and also returns None.Types of these two functions are, respectively:Callable[[Animal], None]Callable[[Dog], None]Now, let’s try to assign dog_run to animal_run, like we (successfully) assigned a_dog to an_animal:Mypy is not happy, interesting. Yet…Why is there an error in the first case but no errors in the second?Also me at work. [source]Subtyping of FunctionsIt seems that animal_run’s type is a subtype of dog_run’s type, or more generally Callable[[Animal], None] <: Callable[[Dog], None]. And this is indeed the case, no mistake here.Apparently, in case of function’s arguments subtyping works the other way around compared to plain objects: function with more general argument type is a subtype of function with less general argument type:for SubType <: SuperTypeit is true that Callable[[SuperType], None] <: Callable[[SubType], None].(This feature is formally called “contravariance”. I define it in the third part of this blog post series: s02e03.)That basically means that when Callable[[Dog], None] (like dog_run()) function is expected Callable[[Animal], None] (like animal_run()) function is also acceptable, but not the opposite.Let’s confirm it with mypy’s help. I will use both run functions defined earlier:Experiment 1Firstly, I will define a make_dog_run function that takes a Callable[[Dog], None] function as a parameter and uses it on a Dog. Next, I will call it with Callable[[Dog], None] and with Callable[[Animal], None] functions. If Callable[[Animal], None] <: Callable[[Dog], None], then mypy should accept both calls.No errors, as expected.Experiment 2Now, I will define a make_animal_run function that takes a Callable[[Animal], None] function as parameter, and call it with the same two run functions: dog_run and animal_run. If Callable[[Animal], None] <: Callable[[Dog], None], then mypy should reject calling make_animal_run with Callable[[Dog], Any] function.Passing dog_run to make_animal_runis rejected by mypy, also as expected.Both experiments show that, in fact, Callable[[Animal], None] <: Callable[[Dog], None] is the case, at least according to mypy. And it really makes sense if you think about it. We should not call make_animal_run with a dog_run. Why?an_animal, on which we call the passed run_dog function, might not be a Dog and run_dog might not be suitable for it. Let’s imagine this:Now, we passed an Animal (every Kangaroo is an Animal) to make_animal_run along with dog_run. And inside make_animal_run, we are effectively forcing a kangaroo to run like a dog. It is so silly even mypy knows it 🙃ResultsThe Original Code Is Not SafeFor convenience, I repeat the code we started with:Let’s see how the original mypy error was not only pointing out some “theoretical inconsistencies” but also a real buggy exploit in our code. This will be very similar to our Kangaroo example, but with classes.Everything might seem fine: Chocolate is a subtype of Food, so I can use it whenever Food is used, like in Animal.eat. Sure, but look at this:This is Lassie. Keep Lassie safe! [source]Lassie is a Dog, which is a subtype of Animal, so I should safely pass her to feed_animal. Can I uncomment the code and run it then? … No! Stop! 🚫 This will poison Lassie! She can only eat Meat (as stated in Dog.eat method) and Chocolate is bad for her. Blindly following badly constructed type relationships, we can harm our dog, or cause other catastrophic results on production.That is why the code with swapped food types (see above) did not trigger any mypy errors. The code cannot be exploited with a function like feed_animal. It cannot because using Animal’s eat to feed a dog cannot force the dog to eat food incompatible with the type of eats’s food parameter, since it’s a general Food.So mypy was really onto something real. It wasn’t just “theoretical”, meaning unpractical. It was “theoretical” in a good way. A good theory has an application to practice, and Python typing theory is a good theory. Let’s get back to it for a moment to pin things down.What Is the “Incompatibility”Let’s remind ourselves subtyping definition. From the definition, it follows that:the set of values becomes smaller in the process of subtyping, while the set of functions becomes larger.So every function from Animal is also in the set of functions from Dog: if Animal can do something, we can be sure that Dog also can do it (by inheritance). Therefore, we should safely use everyAnimal's method/function for a Dog instance. This is not true the other way around: Animal cannot do everything that Dog can. For instance, Animal cannot bark. And it works, as we could see, outside of a class context as well — both animal_run and dog_run are standalone functions. It’s not surprising, since methods are just functions with the first parameter fixed to self (which has a fixed type of the class the method is attached to). Every function/method defined on SuperType objects should be applicable to SubType objects, and it wasn’t true in both kangaroo-runs-like-a-dog and dog-eats-chocolate cases.So, by “incompatible” mypy meant that relations between the types of both eat methods were contradictory:From the inheritance point of view, type of Dog.eat() is a subtype of Animal.eat()(since Dog is a subtype of Animal).At the same time, from the type-of-arguments point of view, Dog.eat() is a supertype of Animal.eat() (as we have seen above).For two distinct A and B types it cannot be that A <: B and B <: A at the same time, so there is a contradiction, and mypy reports it.Why the Problem Was Not That StraightforwardInitially, for me the problem wasn’t easy to grasp. I needed to do some research and to experiment a bit. Everything clicked in place only after I saw an example of code similar to the one with feed_animal function. Why wasn’t the issue more straightforward?I think, in my case, it was a mix of these three reasons:The code looked fine. The code in question, at the first glance, looked correct. Also, it did work in every case, so far. Moreso, it seemed to be confirmed by multiple tests. In cases like this, we sometimes turn off any additional reasoning about the code. We think that the code lacks nothing and we just move on. If the code is simple, we are even more inclined to do so.Class relationships were deceptive. The relationships between classes were somewhat deceptive and led to inaccurate intuitions. We intuitively associate meat with dogs, and type annotations seemed to support this. Yet, in terms of types, Meat class isn’t really related to Dog class, whatsoever. And, as we saw, nothing prevents us from feeding our dog with food different than Meat.I had too little experience. In many Python projects I’ve worked on, class relationships were fairly simple. Most of the time, complex class relationships weren’t needed, as those were Django-based projects. And let’s face it: in many Python projects, the framework takes care of all the structures, and we just fill in the gaps. When we are faced with a more complicated system, especially if it’s not strongly coupled with a web framework, we are forced to learn patterns we’ve never been exposed to before. This makes us develop new intuitions.