@spacegreg posted a new topic, subject "Weird overlay behavior with python packages". It read:

   Recently I've encountered a problem with my personal overlay containing custom versions of Python packages from nixpkgs. When using two overlays for a package (replacing the existing package with my version; overriding some options of that package), it seems like the package entry “resets” to the nixpkgs version. Curiosly, this does not happen if a package is in pkgs root.
    
    Here is a simplified example.
    
    ###  Customized packages:
    
    `custom_hello.nix`:
    [code] 
        { stdenv, optionA ? false, optionB ? false }:
        
        stdenv.mkDerivation {
          pname = "hello";
        
          passthru = {
            inherit optionA optionB;
          };
        }
        
    [/code]
    
    `custom_numpy.nix`:
    [code] 
        { buildPythonPackage, optionA ? false, optionB ? false }:
        
        buildPythonPackage {
          pname = "numpy";
        
          passthru = {
            inherit optionA optionB;
          };
        }
        
    [/code]
    
    ###  Overlays:
    
    `overlays_hello.nix`:
    [code] 
        {
          overlay1 = self: super: with super; {
            hello = callPackage ./custom_hello.nix {};
          };
        
          overlay2 = self: super: {
            hello = super.hello.override { optionA = true; };
          };
        }
        
    [/code]
    
    `overlays_numpy.nix`:
    [code] 
        {
          overlay1 = self: super: {
            python3 = super.python3.override {
              packageOverrides = pself: psuper: with super.python3.pkgs; {
                numpy = callPackage ./custom_hello.nix {};
              };
            };
          };
        
          overlay2 = self: super: {
            python3 = super.python3.override {
              packageOverrides = pself: psuper: {
                numpy = psuper.numpy.override { optionA = true; };
              };
            };
          };
        }
        
    [/code]
    
    ###  In nix-repl
    [code] 
        nix-repl> o_h = import ./overlays_hello.nix 
        nix-repl> o_n = import ./overlays_numpy.nix
        # "Root" package
        nix-repl> pkgs = import <nixpkgs> { overlays = [ o_h.overlay1 ]; } 
        nix-repl> pkgs.hello.optionA                                       
        false
        nix-repl> pkgs = import <nixpkgs> { overlays = [ o_h.overlay1 o_h.overlay2 ]; }
        nix-repl> pkgs.hello.optionA                                                    
        true
        # All good!
        # Python package
        nix-repl> pkgs = import <nixpkgs> { overlays = [ o_n.overlay1 ]; }              
        nix-repl> pkgs.python3Packages.numpy.optionA
        false
        nix-repl> pkgs = import <nixpkgs> { overlays = [ o_n.overlay1 o_n.overlay2 ]; }
        nix-repl> pkgs.python3Packages.numpy.optionA                                    
        error: anonymous function at /nix/store/ifg0cf324ylhvicadnh3pq0rzn8p8n9w-nixos-21.05.2132.733682c3292/nixos/pkgs/development/python-modules/numpy/default.nix:1:1 called with unexpected argument 'optionA', at /nix/store/ifg0cf324ylhvicadnh3pq0rzn8p8n9w-nixos-21.05.2132.733682c3292/nixos/lib/customisation.nix:69:16
        # :(
        
    [/code]
    
    Am I missing something here? Is it a bug?

@FRidh (Frederik Rietdijk) replied

    spacegreg:
    
    > 
    [code]
    >     {
    >       overlay1 = self: super: rec {
    >         python3 = super.python3.override {
    >           self = python3; #
    >           packageOverrides = pself: psuper: with super.python3.pkgs; {
    >             numpy = callPackage ./custom_hello.nix {};
    >           };
    >         };
    >       };
    >     
    >       overlay2 = self: super: {
    >         python3 = super.python3.override {
    >           self = python3; #
    >           packageOverrides = pself: psuper: {
    >             numpy = psuper.numpy.override { optionA = true; };
    >           };
    >         };
    >       };
    >     }
    >     
    [/code]
    
    Common mistake, you need to pass in `self`.

To which @spacegreg replied

    Thanks for the answer. Unfortunately, it didn't work in my case. I used the following `overlays_numpy.nix`:
    [code] 
        {
          overlay1 = self: super: rec {
            python3 = super.python3.override {
              self = python3;
              packageOverrides = pself: psuper: with super.python3.pkgs; {
                numpy = callPackage ./custom_numpy.nix {};
              };
            };
          };
        
          overlay2 = self: super: rec {
            python3 = super.python3.override {
              self = python3;
              packageOverrides = pself: psuper: {
                numpy = psuper.numpy.override { optionA = true; };
              };
            };
          };
        }
        
    [/code]
    
    The same error:
    [code] 
        nix-repl> o_n = import ./overlays_numpy.nix                                     
        
        nix-repl> pkgs = import <nixpkgs> { overlays = [ o_n.overlay1 o_n.overlay2 ]; }
        
        nix-repl> pkgs.python3Packages.numpy.optionA                                    
        error: anonymous function at /nix/store/ifg0cf324ylhvicadnh3pq0rzn8p8n9w-nixos-21.05.2132.733682c3292/nixos/pkgs/development/python-modules/numpy/default.nix:1:1 called with unexpected argument 'optionA', at /nix/store/ifg0cf324ylhvicadnh3pq0rzn8p8n9w-nixos-21.05.2132.733682c3292/nixos/lib/customisation.nix:69:16
        
    [/code]
    
    In this thread it is mentioned that `python.override` breaks if it is used in more than one overlay. Is it true? If it is, how to work around this situation?

To which @spacegreg replied

    After some more experimentation it seems to me that when overriding python package set the original Python package set is always used as “super”. E.g., newly added packages via python.override disappear even if `python = python.override { self = python; packageOverrides self: super: {}; };`. Here is an example:
    [code] 
        let
          pkgs = import <nixpkgs> {};
        
          testPkg = ({ buildPythonPackage, someOption ? false }:
            buildPythonPackage {
              pname = "my-test";
              version = "0.1";
              passthru = { inherit someOption; };
          });
        
          # First override
          python3_1 = pkgs.python3.override {
            self = python3_1;
            packageOverrides = self: super: {
              my-test = super.pkgs.callPackage testPkg { inherit (super) buildPythonPackage; };
            };
          };
        
          # Second override
          python3_2 = python3_1.override {
            self = python3_2;
            packageOverrides = self: super: {
              my-test = super.my-test.override { someOption = true; };
            };
          };
        
        in builtins.trace python3_2.pkgs.my-test.someOption derivation { }
        
    [/code]
    
    While calling `nix-build --no-out-link multi_python_overrides.nix` it returns not this:
    [code] 
        trace: true
        error: derivation name missing
        
    [/code]
    
    but this
    [code] 
        error: attribute 'my-test' missing, at multi_python_overrides.nix:23:17
        
    [/code]
    
    If package override function returns an empty attrset, the error is about missing `my-part` attribute in `python3_2.pkgs`. Trying to modify only the package set as per manual (`myPythonPackages = pkgs.pythonPackages.override { overrides = self: super: {...}; };`), the same problem arises.
    
    I've searched the Nixpkgs docs and haven't found any clue about what I'm missing. I haven't seen any examples of multiple overrides though.

Then @fricklerhandwerk (Valentin Gagarin) said

    Essentially in the current state of things, Python overrides do not compose. I don't have my browser history at hand, but if you search for issues and PRs on GitHub you will find multiple unfinished attempts to rework this.

@spacegreg replied

    Thanks a lot for the info. Well, it's quite unfortunate, and I don't feel knowledgeable enough of Nix/Nixpkgs to do a proper rework myself. I'll just try to hack a solution that would at least work in my specific case.

@FRidh (Frederik Rietdijk) replied

    Indeed, that's Python's packageOverrides isn't composible · Issue #44426 · NixOS/nixpkgs · GitHub.


{'source': 'discourse', 'url': 'https://discourse.nixos.org/t/weird-overlay-behavior-with-python-packages/14469', 'title': 'Weird overlay behavior with python packages', 'views': 362, 'has_accepted_answer': True, 'word_count': 811, 'like_count': 2, 'reply_count': 4, 'participant_count': 3, 'posts': 7, 'tags': '', 'author': '@spacegreg', 'participants': '@spacegreg, @FRidh (Frederik Rietdijk), @fricklerhandwerk (Valentin Gagarin)', 'image_url': [None], 'created_at': '2021-08-09T08:59:26.984Z', 'last_posted_at': '2021-08-12T12:54:05.396Z'}