Chapter 14. Pruning objects to reclaim resources
Chapter 14. Pruning objects to reclaim resources Over time, API objects created in OpenShift Container Platform can accumulate in the cluster's etcd data store through normal user operations, such as when building and deploying applications. Cluster administrators can periodically prune older versions of objects from the cluster that are no longer required. For example, by pruning images you can delete older images and layers that are no longer in use, but are still taking up disk space. 14.1. Basic pruning operations The CLI groups prune operations under a common parent command: USD oc adm prune <object_type> <options> This specifies: The <object_type> to perform the action on, such as groups , builds , deployments , or images . The <options> supported to prune that object type. 14.2. Pruning groups To prune groups records from an external provider, administrators can run the following command: USD oc adm prune groups \ --sync-config=path/to/sync/config [<options>] Table 14.1. oc adm prune groups flags Options Description --confirm Indicate that pruning should occur, instead of performing a dry-run. --blacklist Path to the group blacklist file. --whitelist Path to the group whitelist file. --sync-config Path to the synchronization configuration file. Procedure To see the groups that the prune command deletes, run the following command: USD oc adm prune groups --sync-config=ldap-sync-config.yaml To perform the prune operation, add the --confirm flag: USD oc adm prune groups --sync-config=ldap-sync-config.yaml --confirm 14.3. Pruning deployment resources You can prune resources associated with deployments that are no longer required by the system, due to age and status. The following command prunes replication controllers associated with DeploymentConfig objects: USD oc adm prune deployments [<options>] Note To also prune replica sets associated with Deployment objects, use the --replica-sets flag. This flag is currently a Technology Preview feature. Table 14.2. oc adm prune deployments flags Option Description --confirm Indicate that pruning should occur, instead of performing a dry-run. --keep-complete=<N> Per the DeploymentConfig object, keep the last N replication controllers that have a status of Complete and replica count of zero. The default is 5 . --keep-failed=<N> Per the DeploymentConfig object, keep the last N replication controllers that have a status of Failed and replica count of zero. The default is 1 . --keep-younger-than=<duration> Do not prune any replication controller that is younger than <duration> relative to the current time. Valid units of measurement include nanoseconds ( ns ), microseconds ( us ), milliseconds ( ms ), seconds ( s ), minutes ( m ), and hours ( h ). The default is 60m . --orphans Prune all replication controllers that no longer have a DeploymentConfig object, has status of Complete or Failed , and has a replica count of zero. --replica-sets=true|false If true , replica sets are included in the pruning process. The default is false . Important This flag is a Technology Preview feature. Procedure To see what a pruning operation would delete, run the following command: USD oc adm prune deployments --orphans --keep-complete=5 --keep-failed=1 \ --keep-younger-than=60m To actually perform the prune operation, add the --confirm flag: USD oc adm prune deployments --orphans --keep-complete=5 --keep-failed=1 \ --keep-younger-than=60m --confirm 14.4. Pruning builds To prune builds that are no longer required by the system due to age and status, administrators can run the following command: USD oc adm prune builds [<options>] Table 14.3. oc adm prune builds flags Option Description --confirm Indicate that pruning should occur, instead of performing a dry-run. --orphans Prune all builds whose build configuration no longer exists, status is complete, failed, error, or canceled. --keep-complete=<N> Per build configuration, keep the last N builds whose status is complete. The default is 5 . --keep-failed=<N> Per build configuration, keep the last N builds whose status is failed, error, or canceled. The default is 1 . --keep-younger-than=<duration> Do not prune any object that is younger than <duration> relative to the current time. The default is 60m . Procedure To see what a pruning operation would delete, run the following command: USD oc adm prune builds --orphans --keep-complete=5 --keep-failed=1 \ --keep-younger-than=60m To actually perform the prune operation, add the --confirm flag: USD oc adm prune builds --orphans --keep-complete=5 --keep-failed=1 \ --keep-younger-than=60m --confirm Note Developers can enable automatic build pruning by modifying their build configuration. Additional resources Performing advanced builds Pruning builds 14.5. Automatically pruning images Images from the OpenShift image registry that are no longer required by the system due to age, status, or exceed limits are automatically pruned. Cluster administrators can configure the Pruning Custom Resource, or suspend it. Prerequisites You have access to an OpenShift Container Platform cluster using an account with cluster administrator permissions. Install the oc CLI. Procedure Verify that the object named imagepruners.imageregistry.operator.openshift.io/cluster contains the following spec and status fields: spec: schedule: 0 0 * * * 1 suspend: false 2 keepTagRevisions: 3 3 keepYoungerThanDuration: 60m 4 keepYoungerThan: 3600000000000 5 resources: {} 6 affinity: {} 7 nodeSelector: {} 8 tolerations: [] 9 successfulJobsHistoryLimit: 3 10 failedJobsHistoryLimit: 3 11 status: observedGeneration: 2 12 conditions: 13 - type: Available status: "True" lastTransitionTime: 2019-10-09T03:13:45 reason: Ready message: "Periodic image pruner has been created." - type: Scheduled status: "True" lastTransitionTime: 2019-10-09T03:13:45 reason: Scheduled message: "Image pruner job has been scheduled." - type: Failed staus: "False" lastTransitionTime: 2019-10-09T03:13:45 reason: Succeeded message: "Most recent image pruning job succeeded." 1 schedule : CronJob formatted schedule. This is an optional field, default is daily at midnight. 2 suspend : If set to true , the CronJob running pruning is suspended. This is an optional field, default is false . The initial value on new clusters is false . 3 keepTagRevisions : The number of revisions per tag to keep. This is an optional field, default is 3 . The initial value is 3 . 4 keepYoungerThanDuration : Retain images younger than this duration. This is an optional field. If a value is not specified, either keepYoungerThan or the default value 60m (60 minutes) is used. 5 keepYoungerThan : Deprecated. The same as keepYoungerThanDuration , but the duration is specified as an integer in nanoseconds. This is an optional field. When keepYoungerThanDuration is set, this field is ignored. 6 resources : Standard pod resource requests and limits. This is an optional field. 7 affinity : Standard pod affinity. This is an optional field. 8 nodeSelector : Standard pod node selector. This is an optional field. 9 tolerations : Standard pod tolerations. This is an optional field. 10 successfulJobsHistoryLimit : The maximum number of successful jobs to retain. Must be >= 1 to ensure metrics are reported. This is an optional field, default is 3 . The initial value is 3 . 11 failedJobsHistoryLimit : The maximum number of failed jobs to retain. Must be >= 1 to ensure metrics are reported. This is an optional field, default is 3 . The initial value is 3 . 12 observedGeneration : The generation observed by the Operator. 13 conditions : The standard condition objects with the following types: Available : Indicates if the pruning job has been created. Reasons can be Ready or Error. Scheduled : Indicates if the pruning job has been scheduled. Reasons can be Scheduled, Suspended, or Error. Failed : Indicates if the most recent pruning job failed. Important The Image Registry Operator's behavior for managing the pruner is orthogonal to the managementState specified on the Image Registry Operator's ClusterOperator object. If the Image Registry Operator is not in the Managed state, the image pruner can still be configured and managed by the Pruning Custom Resource. However, the managementState of the Image Registry Operator alters the behavior of the deployed image pruner job: Managed : the --prune-registry flag for the image pruner is set to true . Removed : the --prune-registry flag for the image pruner is set to false , meaning it only prunes image metadata in etcd. 14.6. Manually pruning images The pruning custom resource enables automatic image pruning for the images from the OpenShift image registry. However, administrators can manually prune images that are no longer required by the system due to age, status, or exceed limits. There are two methods to manually prune images: Running image pruning as a Job or CronJob on the cluster. Running the oc adm prune images command. Prerequisites To prune images, you must first log in to the CLI as a user with an access token. The user must also have the system:image-pruner cluster role or greater (for example, cluster-admin ). Expose the image registry. Procedure To manually prune images that are no longer required by the system due to age, status, or exceed limits, use one of the following methods: Run image pruning as a Job or CronJob on the cluster by creating a YAML file for the pruner service account, for example: USD oc create -f <filename>.yaml Example output kind: List apiVersion: v1 items: - apiVersion: v1 kind: ServiceAccount metadata: name: pruner namespace: openshift-image-registry - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: openshift-image-registry-pruner roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:image-pruner subjects: - kind: ServiceAccount name: pruner namespace: openshift-image-registry - apiVersion: batch/v1 kind: CronJob metadata: name: image-pruner namespace: openshift-image-registry spec: schedule: "0 0 * * *" concurrencyPolicy: Forbid successfulJobsHistoryLimit: 1 failedJobsHistoryLimit: 3 jobTemplate: spec: template: spec: restartPolicy: OnFailure containers: - image: "quay.io/openshift/origin-cli:4.1" resources: requests: cpu: 1 memory: 1Gi terminationMessagePolicy: FallbackToLogsOnError command: - oc args: - adm - prune - images - --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt - --keep-tag-revisions=5 - --keep-younger-than=96h - --confirm=true name: image-pruner serviceAccountName: pruner Run the oc adm prune images [<options>] command: USD oc adm prune images [<options>] Pruning images removes data from the integrated registry unless --prune-registry=false is used. Pruning images with the --namespace flag does not remove images, only image streams. Images are non-namespaced resources. Therefore, limiting pruning to a particular namespace makes it impossible to calculate its current usage. By default, the integrated registry caches metadata of blobs to reduce the number of requests to storage, and to increase the request-processing speed. Pruning does not update the integrated registry cache. Images that still contain pruned layers after pruning will be broken because the pruned layers that have metadata in the cache will not be pushed. Therefore, you must redeploy the registry to clear the cache after pruning: USD oc rollout restart deployment/image-registry -n openshift-image-registry If the integrated registry uses a Redis cache, you must clean the database manually. If redeploying the registry after pruning is not an option, then you must permanently disable the cache. oc adm prune images operations require a route for your registry. Registry routes are not created by default. The Prune images CLI configuration options table describes the options you can use with the oc adm prune images <options> command. Table 14.4. Prune images CLI configuration options Option Description --all Include images that were not pushed to the registry, but have been mirrored by pullthrough. This is on by default. To limit the pruning to images that were pushed to the integrated registry, pass --all=false . --certificate-authority The path to a certificate authority file to use when communicating with the OpenShift Container Platform-managed registries. Defaults to the certificate authority data from the current user's configuration file. If provided, a secure connection is initiated. --confirm Indicate that pruning should occur, instead of performing a test-run. This requires a valid route to the integrated container image registry. If this command is run outside of the cluster network, the route must be provided using --registry-url . --force-insecure Use caution with this option. Allow an insecure connection to the container registry that is hosted via HTTP or has an invalid HTTPS certificate. --keep-tag-revisions=<N> For each imagestream, keep up to at most N image revisions per tag (default 3 ). --keep-younger-than=<duration> Do not prune any image that is younger than <duration> relative to the current time. Alternately, do not prune any image that is referenced by any other object that is younger than <duration> relative to the current time (default 60m ). --prune-over-size-limit Prune each image that exceeds the smallest limit defined in the same project. This flag cannot be combined with --keep-tag-revisions nor --keep-younger-than . --registry-url The address to use when contacting the registry. The command attempts to use a cluster-internal URL determined from managed images and image streams. In case it fails (the registry cannot be resolved or reached), an alternative route that works needs to be provided using this flag. The registry hostname can be prefixed by https:// or http:// , which enforces particular connection protocol. --prune-registry In conjunction with the conditions stipulated by the other options, this option controls whether the data in the registry corresponding to the OpenShift Container Platform image API object is pruned. By default, image pruning processes both the image API objects and corresponding data in the registry. This option is useful when you are only concerned with removing etcd content, to reduce the number of image objects but are not concerned with cleaning up registry storage, or if you intend to do that separately by hard pruning the registry during an appropriate maintenance window for the registry. 14.6.1. Image prune conditions You can apply conditions to your manually pruned images. To remove any image managed by OpenShift Container Platform, or images with the annotation openshift.io/image.managed : Created at least --keep-younger-than minutes ago and are not currently referenced by any: Pods created less than --keep-younger-than minutes ago Image streams created less than --keep-younger-than minutes ago Running pods Pending pods Replication controllers Deployments Deployment configs Replica sets Build configurations Builds Jobs Cronjobs Stateful sets --keep-tag-revisions most recent items in stream.status.tags[].items That are exceeding the smallest limit defined in the same project and are not currently referenced by any: Running pods Pending pods Replication controllers Deployments Deployment configs Replica sets Build configurations Builds Jobs Cronjobs Stateful sets There is no support for pruning from external registries. When an image is pruned, all references to the image are removed from all image streams that have a reference to the image in status.tags . Image layers that are no longer referenced by any images are removed. Note The --prune-over-size-limit flag cannot be combined with the --keep-tag-revisions flag nor the --keep-younger-than flags. Doing so returns information that this operation is not allowed. Separating the removal of OpenShift Container Platform image API objects and image data from the registry by using --prune-registry=false , followed by hard pruning the registry, can narrow timing windows and is safer when compared to trying to prune both through one command. However, timing windows are not completely removed. For example, you can still create a pod referencing an image as pruning identifies that image for pruning. You should still keep track of an API object created during the pruning operations that might reference images so that you can mitigate any references to deleted content. Re-doing the pruning without the --prune-registry option or with --prune-registry=true does not lead to pruning the associated storage in the image registry for images previously pruned by --prune-registry=false . Any images that were pruned with --prune-registry=false can only be deleted from registry storage by hard pruning the registry. 14.6.2. Running the image prune operation Procedure To see what a pruning operation would delete: Keeping up to three tag revisions, and keeping resources (images, image streams, and pods) younger than 60 minutes: USD oc adm prune images --keep-tag-revisions=3 --keep-younger-than=60m Pruning every image that exceeds defined limits: USD oc adm prune images --prune-over-size-limit To perform the prune operation with the options from the step: USD oc adm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm USD oc adm prune images --prune-over-size-limit --confirm 14.6.3. Using secure or insecure connections The secure connection is the preferred and recommended approach. It is done over HTTPS protocol with a mandatory certificate verification. The prune command always attempts to use it if possible. If it is not possible, in some cases it can fall-back to insecure connection, which is dangerous. In this case, either certificate verification is skipped or plain HTTP protocol is used. The fall-back to insecure connection is allowed in the following cases unless --certificate-authority is specified: The prune command is run with the --force-insecure option. The provided registry-url is prefixed with the http:// scheme. The provided registry-url is a local-link address or localhost . The configuration of the current user allows for an insecure connection. This can be caused by the user either logging in using --insecure-skip-tls-verify or choosing the insecure connection when prompted. Important If the registry is secured by a certificate authority different from the one used by OpenShift Container Platform, it must be specified using the --certificate-authority flag. Otherwise, the prune command fails with an error. 14.6.4. Image pruning problems Images not being pruned If your images keep accumulating and the prune command removes just a small portion of what you expect, ensure that you understand the image prune conditions that must apply for an image to be considered a candidate for pruning. Ensure that images you want removed occur at higher positions in each tag history than your chosen tag revisions threshold. For example, consider an old and obsolete image named sha256:abz . By running the following command in your namespace, where the image is tagged, the image is tagged three times in a single image stream named myapp : USD oc get is -n <namespace> -o go-template='{{range USDisi, USDis := .items}}{{range USDti, USDtag := USDis.status.tags}}'\ '{{range USDii, USDitem := USDtag.items}}{{if eq USDitem.image "sha256:<hash>"}}{{USDis.metadata.name}}:{{USDtag.tag}} at position {{USDii}} out of {{len USDtag.items}}\n'\ '{{end}}{{end}}{{end}}{{end}}' Example output myapp:v2 at position 4 out of 5 myapp:v2.1 at position 2 out of 2 myapp:v2.1-may-2016 at position 0 out of 1 When default options are used, the image is never pruned because it occurs at position 0 in a history of myapp:v2.1-may-2016 tag. For an image to be considered for pruning, the administrator must either: Specify --keep-tag-revisions=0 with the oc adm prune images command. Warning This action removes all the tags from all the namespaces with underlying images, unless they are younger or they are referenced by objects younger than the specified threshold. Delete all the istags where the position is below the revision threshold, which means myapp:v2.1 and myapp:v2.1-may-2016 . Move the image further in the history, either by running new builds pushing to the same istag , or by tagging other image. This is not always desirable for old release tags. Tags having a date or time of a particular image's build in their names should be avoided, unless the image must be preserved for an undefined amount of time. Such tags tend to have just one image in their history, which prevents them from ever being pruned. Using a secure connection against insecure registry If you see a message similar to the following in the output of the oc adm prune images command, then your registry is not secured and the oc adm prune images client attempts to use a secure connection: error: error communicating with registry: Get https://172.30.30.30:5000/healthz: http: server gave HTTP response to HTTPS client The recommended solution is to secure the registry. Otherwise, you can force the client to use an insecure connection by appending --force-insecure to the command; however, this is not recommended. Using an insecure connection against a secured registry If you see one of the following errors in the output of the oc adm prune images command, it means that your registry is secured using a certificate signed by a certificate authority other than the one used by oc adm prune images client for connection verification: error: error communicating with registry: Get http://172.30.30.30:5000/healthz: malformed HTTP response "\x15\x03\x01\x00\x02\x02" error: error communicating with registry: [Get https://172.30.30.30:5000/healthz: x509: certificate signed by unknown authority, Get http://172.30.30.30:5000/healthz: malformed HTTP response "\x15\x03\x01\x00\x02\x02"] By default, the certificate authority data stored in the user's configuration files is used; the same is true for communication with the master API. Use the --certificate-authority option to provide the right certificate authority for the container image registry server. Using the wrong certificate authority The following error means that the certificate authority used to sign the certificate of the secured container image registry is different from the authority used by the client: error: error communicating with registry: Get https://172.30.30.30:5000/: x509: certificate signed by unknown authority Make sure to provide the right one with the flag --certificate-authority . As a workaround, the --force-insecure flag can be added instead. However, this is not recommended. Additional resources Accessing the registry Exposing the registry See Image Registry Operator in OpenShift Container Platform for information on how to create a registry route. 14.7. Hard pruning the registry The OpenShift Container Registry can accumulate blobs that are not referenced by the OpenShift Container Platform cluster's etcd. The basic pruning images procedure, therefore, is unable to operate on them. These are called orphaned blobs . Orphaned blobs can occur from the following scenarios: Manually deleting an image with oc delete image <sha256:image-id> command, which only removes the image from etcd, but not from the registry's storage. Pushing to the registry initiated by daemon failures, which causes some blobs to get uploaded, but the image manifest (which is uploaded as the very last component) does not. All unique image blobs become orphans. OpenShift Container Platform refusing an image because of quota restrictions. The standard image pruner deleting an image manifest, but is interrupted before it deletes the related blobs. A bug in the registry pruner, which fails to remove the intended blobs, causing the image objects referencing them to be removed and the blobs becoming orphans. Hard pruning the registry, a separate procedure from basic image pruning, allows cluster administrators to remove orphaned blobs. You should hard prune if you are running out of storage space in your OpenShift Container Registry and believe you have orphaned blobs. This should be an infrequent operation and is necessary only when you have evidence that significant numbers of new orphans have been created. Otherwise, you can perform standard image pruning at regular intervals, for example, once a day (depending on the number of images being created). Procedure To hard prune orphaned blobs from the registry: Log in. Log in to the cluster with the CLI as kubeadmin or another privileged user that has access to the openshift-image-registry namespace. Run a basic image prune . Basic image pruning removes additional images that are no longer needed. The hard prune does not remove images on its own. It only removes blobs stored in the registry storage. Therefore, you should run this just before the hard prune. Switch the registry to read-only mode. If the registry is not running in read-only mode, any pushes happening at the same time as the prune will either: fail and cause new orphans, or succeed although the images cannot be pulled (because some of the referenced blobs were deleted). Pushes will not succeed until the registry is switched back to read-write mode. Therefore, the hard prune must be carefully scheduled. To switch the registry to read-only mode: In configs.imageregistry.operator.openshift.io/cluster , set spec.readOnly to true : USD oc patch configs.imageregistry.operator.openshift.io/cluster -p '{"spec":{"readOnly":true}}' --type=merge Add the system:image-pruner role. The service account used to run the registry instances requires additional permissions to list some resources. Get the service account name: USD service_account=USD(oc get -n openshift-image-registry \ -o jsonpath='{.spec.template.spec.serviceAccountName}' deploy/image-registry) Add the system:image-pruner cluster role to the service account: USD oc adm policy add-cluster-role-to-user \ system:image-pruner -z \ USD{service_account} -n openshift-image-registry Optional: Run the pruner in dry-run mode. To see how many blobs would be removed, run the hard pruner in dry-run mode. No changes are actually made. The following example references an image registry pod called image-registry-3-vhndw : USD oc -n openshift-image-registry exec pod/image-registry-3-vhndw -- /bin/sh -c '/usr/bin/dockerregistry -prune=check' Alternatively, to get the exact paths for the prune candidates, increase the logging level: USD oc -n openshift-image-registry exec pod/image-registry-3-vhndw -- /bin/sh -c 'REGISTRY_LOG_LEVEL=info /usr/bin/dockerregistry -prune=check' Example output time="2017-06-22T11:50:25.066156047Z" level=info msg="start prune (dry-run mode)" distribution_version="v2.4.1+unknown" kubernetes_version=v1.6.1+USDFormat:%hUSD openshift_version=unknown time="2017-06-22T11:50:25.092257421Z" level=info msg="Would delete blob: sha256:00043a2a5e384f6b59ab17e2c3d3a3d0a7de01b2cabeb606243e468acc663fa5" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:25.092395621Z" level=info msg="Would delete blob: sha256:0022d49612807cb348cabc562c072ef34d756adfe0100a61952cbcb87ee6578a" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:25.092492183Z" level=info msg="Would delete blob: sha256:0029dd4228961086707e53b881e25eba0564fa80033fbbb2e27847a28d16a37c" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:26.673946639Z" level=info msg="Would delete blob: sha256:ff7664dfc213d6cc60fd5c5f5bb00a7bf4a687e18e1df12d349a1d07b2cf7663" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:26.674024531Z" level=info msg="Would delete blob: sha256:ff7a933178ccd931f4b5f40f9f19a65be5eeeec207e4fad2a5bafd28afbef57e" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:26.674675469Z" level=info msg="Would delete blob: sha256:ff9b8956794b426cc80bb49a604a0b24a1553aae96b930c6919a6675db3d5e06" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 ... Would delete 13374 blobs Would free up 2.835 GiB of disk space Use -prune=delete to actually delete the data Run the hard prune. Execute the following command inside one running instance of a image-registry pod to run the hard prune. The following example references an image registry pod called image-registry-3-vhndw : USD oc -n openshift-image-registry exec pod/image-registry-3-vhndw -- /bin/sh -c '/usr/bin/dockerregistry -prune=delete' Example output Deleted 13374 blobs Freed up 2.835 GiB of disk space Switch the registry back to read-write mode. After the prune is finished, the registry can be switched back to read-write mode. In configs.imageregistry.operator.openshift.io/cluster , set spec.readOnly to false : USD oc patch configs.imageregistry.operator.openshift.io/cluster -p '{"spec":{"readOnly":false}}' --type=merge 14.8. Pruning cron jobs Cron jobs can perform pruning of successful jobs, but might not properly handle failed jobs. Therefore, the cluster administrator should perform regular cleanup of jobs manually. They should also restrict the access to cron jobs to a small group of trusted users and set appropriate quota to prevent the cron job from creating too many jobs and pods. Additional resources Running tasks in pods using jobs Resource quotas across multiple projects Using RBAC to define and apply permissions
oc adm prune <object_type> <options> oc adm prune groups --sync-config=path/to/sync/config [<options>] oc adm prune groups --sync-config=ldap-sync-config.yaml oc adm prune groups --sync-config=ldap-sync-config.yaml --confirm oc adm prune deployments [<options>] oc adm prune deployments --orphans --keep-complete=5 --keep-failed=1 --keep-younger-than=60m oc adm prune deployments --orphans --keep-complete=5 --keep-failed=1 --keep-younger-than=60m --confirm oc adm prune builds [<options>] oc adm prune builds --orphans --keep-complete=5 --keep-failed=1 --keep-younger-than=60m oc adm prune builds --orphans --keep-complete=5 --keep-failed=1 --keep-younger-than=60m --confirm spec: schedule: 0 0 * * * 1 suspend: false 2 keepTagRevisions: 3 3 keepYoungerThanDuration: 60m 4 keepYoungerThan: 3600000000000 5 resources: {} 6 affinity: {} 7 nodeSelector: {} 8 tolerations: [] 9 successfulJobsHistoryLimit: 3 10 failedJobsHistoryLimit: 3 11 status: observedGeneration: 2 12 conditions: 13 - type: Available status: "True" lastTransitionTime: 2019-10-09T03:13:45 reason: Ready message: "Periodic image pruner has been created." - type: Scheduled status: "True" lastTransitionTime: 2019-10-09T03:13:45 reason: Scheduled message: "Image pruner job has been scheduled." - type: Failed staus: "False" lastTransitionTime: 2019-10-09T03:13:45 reason: Succeeded message: "Most recent image pruning job succeeded." oc create -f <filename>.yaml kind: List apiVersion: v1 items: - apiVersion: v1 kind: ServiceAccount metadata: name: pruner namespace: openshift-image-registry - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: openshift-image-registry-pruner roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:image-pruner subjects: - kind: ServiceAccount name: pruner namespace: openshift-image-registry - apiVersion: batch/v1 kind: CronJob metadata: name: image-pruner namespace: openshift-image-registry spec: schedule: "0 0 * * *" concurrencyPolicy: Forbid successfulJobsHistoryLimit: 1 failedJobsHistoryLimit: 3 jobTemplate: spec: template: spec: restartPolicy: OnFailure containers: - image: "quay.io/openshift/origin-cli:4.1" resources: requests: cpu: 1 memory: 1Gi terminationMessagePolicy: FallbackToLogsOnError command: - oc args: - adm - prune - images - --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt - --keep-tag-revisions=5 - --keep-younger-than=96h - --confirm=true name: image-pruner serviceAccountName: pruner oc adm prune images [<options>] oc rollout restart deployment/image-registry -n openshift-image-registry oc adm prune images --keep-tag-revisions=3 --keep-younger-than=60m oc adm prune images --prune-over-size-limit oc adm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm oc adm prune images --prune-over-size-limit --confirm oc get is -n <namespace> -o go-template='{{range USDisi, USDis := .items}}{{range USDti, USDtag := USDis.status.tags}}' '{{range USDii, USDitem := USDtag.items}}{{if eq USDitem.image "sha256:<hash>"}}{{USDis.metadata.name}}:{{USDtag.tag}} at position {{USDii}} out of {{len USDtag.items}}\n' '{{end}}{{end}}{{end}}{{end}}' myapp:v2 at position 4 out of 5 myapp:v2.1 at position 2 out of 2 myapp:v2.1-may-2016 at position 0 out of 1 error: error communicating with registry: Get https://172.30.30.30:5000/healthz: http: server gave HTTP response to HTTPS client error: error communicating with registry: Get http://172.30.30.30:5000/healthz: malformed HTTP response "\x15\x03\x01\x00\x02\x02" error: error communicating with registry: [Get https://172.30.30.30:5000/healthz: x509: certificate signed by unknown authority, Get http://172.30.30.30:5000/healthz: malformed HTTP response "\x15\x03\x01\x00\x02\x02"] error: error communicating with registry: Get https://172.30.30.30:5000/: x509: certificate signed by unknown authority oc patch configs.imageregistry.operator.openshift.io/cluster -p '{"spec":{"readOnly":true}}' --type=merge service_account=USD(oc get -n openshift-image-registry -o jsonpath='{.spec.template.spec.serviceAccountName}' deploy/image-registry) oc adm policy add-cluster-role-to-user system:image-pruner -z USD{service_account} -n openshift-image-registry oc -n openshift-image-registry exec pod/image-registry-3-vhndw -- /bin/sh -c '/usr/bin/dockerregistry -prune=check' oc -n openshift-image-registry exec pod/image-registry-3-vhndw -- /bin/sh -c 'REGISTRY_LOG_LEVEL=info /usr/bin/dockerregistry -prune=check' time="2017-06-22T11:50:25.066156047Z" level=info msg="start prune (dry-run mode)" distribution_version="v2.4.1+unknown" kubernetes_version=v1.6.1+USDFormat:%hUSD openshift_version=unknown time="2017-06-22T11:50:25.092257421Z" level=info msg="Would delete blob: sha256:00043a2a5e384f6b59ab17e2c3d3a3d0a7de01b2cabeb606243e468acc663fa5" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:25.092395621Z" level=info msg="Would delete blob: sha256:0022d49612807cb348cabc562c072ef34d756adfe0100a61952cbcb87ee6578a" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:25.092492183Z" level=info msg="Would delete blob: sha256:0029dd4228961086707e53b881e25eba0564fa80033fbbb2e27847a28d16a37c" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:26.673946639Z" level=info msg="Would delete blob: sha256:ff7664dfc213d6cc60fd5c5f5bb00a7bf4a687e18e1df12d349a1d07b2cf7663" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:26.674024531Z" level=info msg="Would delete blob: sha256:ff7a933178ccd931f4b5f40f9f19a65be5eeeec207e4fad2a5bafd28afbef57e" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 time="2017-06-22T11:50:26.674675469Z" level=info msg="Would delete blob: sha256:ff9b8956794b426cc80bb49a604a0b24a1553aae96b930c6919a6675db3d5e06" go.version=go1.7.5 instance.id=b097121c-a864-4e0c-ad6c-cc25f8fdf5a6 Would delete 13374 blobs Would free up 2.835 GiB of disk space Use -prune=delete to actually delete the data oc -n openshift-image-registry exec pod/image-registry-3-vhndw -- /bin/sh -c '/usr/bin/dockerregistry -prune=delete' Deleted 13374 blobs Freed up 2.835 GiB of disk space oc patch configs.imageregistry.operator.openshift.io/cluster -p '{"spec":{"readOnly":false}}' --type=merge