Hey just an update on this issue.
As the pinned post above suggested we are removing the ability to be whitelisted to work around these issues (as we need to leave the change in place for security purposes).
I have written this post however to assist you in fixing these issues for good in your builds (so they can continue to run in docker both with and without user namespace remapping).
Changes to make your containers more secure on Bitbucket Pipelines.
Thanks for that link @George Tall, I've run the find command they suggested in both my build container and the actual image I'm building and neither include any files owned by 165586!
This is really frustrating that builds have started failing out of nowhere, and we're seeing the exact same Container ID 165586 error. The CircleCI sol'n did nothing for us either. And our failing images are only ~310MB.
Same issue here. One of our builds fails 100% of the time.
Are you guys confident this is a Bitbucket bug? Have any of you found a workaround?
I've also got a build that fails 100% of the time. The fact that both @Bradley Wogsland and I have Container ID 165586 makes me more suspicious that it's a Bitbucket problem rather than a me problem, though I'm open to being wrong!
Mine fails with a different ID. as per below error message.
There is a workaround described in that issue so please take a look everyone.
I looked at that first - unfortunately I haven't a multistage build, and my build is already running as root! I tried adding a chown for each of the files, but it didn't seem to help. My sense is that this feels like the same issue, but perhaps in a slightly different place, as the workaround doesn't appear to apply.
I'd love to be wrong, but I did try!
I did encounter the problem without multistage build as well. I have not been able to fix the problem yet either.
I have contacted support, who rolled-back the introduction of user mapping for our account. builds are back! This is definitely not a fix, but at least allows to get some work done.
I have also been advised that a guide will soon be published as well to deal with that situation. To be notified, you can "watch" this page.
Thanks @Olivier Lacroix! I'll do the same and keep an eye on that page for future.
Issue #17405 was marked as a duplicate of this issue.
Hope this helps in finding a solution / work around.
As a part of the ongoing security hardening process in Bitbucket Pipelines, we decided to enable docker user namespace remap to eliminate potential security risks that might affect our users.
This change affected a small percentage of users using docker in their builds. So, we wanted to share the different limitations that this change introduced, and the actions required to make your builds work with the security enhancement introduced.
Issues caused by limitations of userns remap.
sharing PID or NET namespaces with the host (--pid=host or --network=host).
external (volume or storage) drivers which are unaware or incapable of using daemon user mappings.
Using the --privileged mode flag on docker run without also specifying --userns=host.
We already disallowed on pipelines --pid=host and --privileged containers however --network=host is a new limitation.
In the interim we have added all users who currently use this feature of docker (as indicated by our analytics) to an exclusion list.
We will be making changes to docker to assist you in moving from using --network=host to a solution that will involve you using container links, port mapping and providing a similar feature to docker for OSX with a host that resolves to the host for the build.
We will release further communications as to when these changes will be made and the exclusion list will be removed.
Docker build, removing files doesn't remove them.
Due to bugs in docker, when files were modified (removed/alterered) during a docker build via RUN/COPY commands the changes weren't being reflected in subsequent layers.
We have upgraded docker to pull in the fix for these issues.
However some failures require user intervention, as some commands were modifying the UID/GID (similar to the error below). For these situations docker now supports --chown as an argument to some dockerfile commands.
Due to the way userns remap works (by remapping UID/GIDs to another less privileged UID/GID in the hosts namespace), UID/GIDs that are placed on files must be in the range 0-65535.
If you recieve this error, you will need to perform a fix on the image that has this invalid UID/GID.
If your recieved this error against a file that you created and placed into an image, the fix is to simply change the UID/GID of the file to one within the range.
To find the file, first get the invalid UID/GID from the error message and run the following command on the files within the container.
If you received this error against a file that was already present in a layer of the base image you depend on, you will need to raise a support case and work with the image maintainers of the base image to get this resolved.
I have the same issue "Error processing tar file(exit status 1): Container ID 165586 cannot be mapped to a host ID" When running python3 pip on requirement file.
Same for me. I have this issue when running python2 pip on requirements file and when 'removing intermediate container'. I tried to change ownership, run as root. The image which I run does not have file with this ID. Weird thing.
For us this was caused by pip install --upgrade pip. Not upgrading pip fixed the build (but may leave you with an old version of pip, of course).
Issue #18071 was marked as a duplicate of this issue.
Hey @Nathan Burrell , would help if you or your team actually read these messages. The "Container ID Cannot Be Mapped to Host ID Error” we are getting is not for a single file, as your pinned post suggests. And simply flagging it as a problem with the base image doesn't help any of us relying on python.
This will probably get worse for python repos as those with pip cache optimizations cycle through caches: I am assuming that the cache feature is unaffected, yet when the cache is out of date or deleted, the image will then be subject to this change.
The solution of finding the files in the image is null for pip installs. There appears to be a process where pip may have intermediate files in userns sensitive areas in the setup process that are then deleted. This makes it impossible to find the files dynamically.
This way /src should already be user mappable, though you may have to chown of the /src directory.
On another branch of this issue: https://github.com/moby/moby/issues/34645 has not been closed. Using multi-stage builds to copy files from docker images is now a broken feature in bitbucket pipelines without hacking around.