Testing docker image builds when low level libraries change Background
In our organization we have many small repos for shared libraries used among our applications. Particularly, we have many Python libraries and Python applications. We have a structures like:

myorg/python-lib1: produces package myorg.lib1
myorg/python-lib2: produces package myorg.lib2
myorg/python-app: produces a package myorg.app, depends on myorg.lib1 and myorg.lib2

In our CI for myorg/python-lib1 and myorg/python-lib1 we run unit tests when PRs come in and publish wheels to PyPI on merges to the main branch.
In our CI for myorg/python-app1 we will build test application images when PRs come, deploy them to a test kubernetes cluster, and allow developers to run integration tests against it. It has a Dockerfile like:
# Builds image reistry.myorg.com/python-app
FROM reistry.myorg.com/python-base:latest

COPY . /app

# Implicitly gets the latest `myorg.lib1` and `myorg.lib2` from PyPI
RUN pip install /app

Note: We also have some projects using build-packs to achieve a similar goal.

Issue
Because we only run the integration tests for myorg/python-app1, we can't get as much confidence when a change to myorg/python-lib1 and myorg/python-lib2 happens. We have to wait for the libraries to get published to PyPI and then rebuild the application images. When there's an issue in the library, this usually involves us git reverting and fixing the PyPI version. This is super cumbersome when we have a lot of images that pull in the dependency before we detect it needs to be rolled back.
I'm looking to restructure the way we build images in myorg/python-app1 (and other apps that share this pattern) to support pulling in dependencies based off of the PR branches.
I've looked at a lot of tools for containerization but I haven't seen any strategies for building application containers given a change in a lower level dependency. I'm interested in seeing what strategies exist for this. Hopefully the strategy could be applied to other languages as well, where we have a similar strategies for applications written in languages like Java, Golang, and npm.
So I believe I came up with a somewhat re-usable solution for this problem using devpi. It allows you to use the same Dockerfile for the testing and production image.
Devpi allows me to create my own index which can shadow my production index with versions of software that haven't been officially released.
Steps to make this work:
1. Base image from ARG
I first edited my Dockerfile to allow overwriting the base image used. This will allow me to create a layer in the middle that configures pip to talk to devpi:
# Builds image reistry.myorg.com/python-app
ARG BASE=reistry.myorg.com/python-base:latest
FROM $BASE

COPY . /app

# Implicitly gets the latest `myorg.lib1` and `myorg.lib2` from PyPI
RUN pip install /app

2. Upload test versions of library
For each python change:

Clone the repo
python -m build the repo
devpi upload the wheels

3. Create a layer with devpi pip conf
Make a pip conf that can access your devpi instance and configure to your index name:
[global]
index-url = http://&lt;some ip&gt;:&lt;some port&gt;/root/dev/+simple/
trusted-host = &lt;some ip&gt;
disable-pip-version-check = true

[search]
index = http://&lt;some ip&gt;:&lt;some port&gt;/root/dev/

And add that to your base images pip conf:
FROM reistry.myorg.com/python-base:latest

COPY pip.conf /etc/pip.conf

4. Chain the images together
docker build -t devpi-base -f devpi.Dockerfile .
docker build -t my-app -f Dockerfile --build-arg=BASE=devpi-base .
docker tag my-app reistry.myorg.com/my-app:test1
docker push reistry.myorg.com/my-app:test1


Now when pip installs all of your dependencies you can have it pick up lower level changes to python packages. This concept should be re-usable for a lot of languages I imagine!