Testing docker image builds when low level libraries change
So I believe I came up with a somewhat re-usable solution for this problem using devpi. It allows you to use the same Dockerfile for the testing and production image.
Devpi allows me to create my own index which can shadow my production index with versions of software that haven't been officially released.
Steps to make this work:
1. Base image from ARG
I first edited my Dockerfile to allow overwriting the base image used. This will allow me to create a layer in the middle that configures pip to talk to devpi:
# Builds image reistry.myorg.com/python-app
ARG BASE=reistry.myorg.com/python-base:latest
FROM $BASE

COPY . /app

# Implicitly gets the latest `myorg.lib1` and `myorg.lib2` from PyPI
RUN pip install /app

2. Upload test versions of library
For each python change:

Clone the repo
python -m build the repo
devpi upload the wheels

3. Create a layer with devpi pip conf
Make a pip conf that can access your devpi instance and configure to your index name:
[global]
index-url = http://&lt;some ip&gt;:&lt;some port&gt;/root/dev/+simple/
trusted-host = &lt;some ip&gt;
disable-pip-version-check = true

[search]
index = http://&lt;some ip&gt;:&lt;some port&gt;/root/dev/

And add that to your base images pip conf:
FROM reistry.myorg.com/python-base:latest

COPY pip.conf /etc/pip.conf

4. Chain the images together
docker build -t devpi-base -f devpi.Dockerfile .
docker build -t my-app -f Dockerfile --build-arg=BASE=devpi-base .
docker tag my-app reistry.myorg.com/my-app:test1
docker push reistry.myorg.com/my-app:test1


Now when pip installs all of your dependencies you can have it pick up lower level changes to python packages. This concept should be re-usable for a lot of languages I imagine!