Prune Kubernetes Resources

Updated 11 hours ago by Michael Cretzman

Currently, this feature is behind a Feature Flag. Contact Harness Support to enable the feature. Feature Flags can only be removed for Harness Professional and Essentials editions. Once the feature is released to a general audience, it's available for Trial and Community Editions.

See New features added to Harness and Features behind Feature Flags (Early Access) for Feature Flag information.

Changes to the manifests used in Harness Kubernetes deployments can result in orphaned resources you are unaware of.

For example, one deployment might deploy resources A and B but the next deployment deploys A and C. C is the new resource and B was removed from the manifest. Without pruning, resource B will remain in the cluster.

You can manually delete Kubernetes resources using the Delete step, but Harness will also perform resource pruning automatically during deployment.

Harness uses pruning by default to remove any resources that were present in an old manifest, but no longer present in the manifest used for the current deployment.

Harness also allows you to identify resources you do not want pruned using the annotation harness.io/skipPruning.

In this topic:

Before You Begin

Supported Platforms and Technologies

Pruning is supported for the following deployment strategies and Workflow types:

  • Rolling Deployments
  • Blue/Green Deployments

See:

Limitations

  • If the resource to be pruned is also part of another Harness Service, Harness does not know this. Be careful when pruning resources and use the annotation harness.io/skipPruning: "true" when in doubt.
  • To prevent pruning using the Harness annotation harness.io/skipPruning: "true", the resource must have been deployed by Harness.
    You cannot add a resource outside of a Harness deployment and apply the annotation and have Harness skip the pruning of that resource.
    If you make any changes to your Kubernetes resources using a tool other than Harness (before or after the deployment), Harness does not track those changes.
  • The maximum manifest/chart size is 0.5MB. When Harness prunes, it stores the full manifest in configMap to use it as part of release history. While deploying very large manifests/charts though Kubernetes, Harness is limited by configMap capacity.

Review: Harness Kubernetes Pruning Criteria

Kubernetes pruning in Harness is the same as the kubectl apply --prune method provided by Kubernetes.

Kubernetes pruning queries the API server for all objects matching a set of labels and attempts to match the returned live object configurations against the object configuration files.

Similarly, Harness compares the objects you are deploying with the objects it finds in the cluster. If Harness finds objects which are not in the current release, it prunes them.

Harness also allows you to identify resources you do not want pruned using the annotation harness.io/skipPruning. This is described later in this topic.

Rolling Deployments

Kubernetes Rolling deployments manage pruning as follows:

In the Deploy stage of the Workflow, Harness compares resources in the last successful release with the current release.

Harness prunes the resources from the last successful release that are not in current release.

If a deployment fails, Harness recreates the pruned resources during its Rollback stage.

During rollback, any new resources that were created in the failed deployment stage that were not in the last successful release are deleted also.

Blue/Green Deployments

Kubernetes Blue/Green deployments manage pruning as follows:

In the first step of a Blue/Green deployment, the new version of the release is deployed to the stage environment (pod set).

Harness prunes by comparing the new and previous releases in the stage pod set. Harness prunes the resources from the last successful release which are not in the current release.

Review: Pruning Examples

The first time you deploy a resource (Deployment, StatefulSet, ReplicaSet, etc) no pruning will take place.

In Harness Deployments, in the Workflow step, you will see a Prune section with the following message:

No previous successful deployment found, so no pruning required

When Harness finds resources that match the pruning criteria, you will see a message like this:

kubectl --kubeconfig=config delete Deployment/k8s-orphaned-resource-b --namespace=default

deployment.apps "k8s-orphaned-resource-b" deleted

kubectl --kubeconfig=config delete ConfigMap/k8s-orphaned-resource-configmap-b --namespace=default

configmap "k8s-orphaned-resource-configmap-b" deleted

Pruning step completed

If a Deployment fails, Harness recreates any of the pruned resources it removed as part of the deployment. In the Rollback Deployment step, you will see a Recreate Pruned Resources section with message like this:

kubectl --kubeconfig=config apply --filename=manifests.yaml --record

deployment.apps/k8s-orphaned-resource-f created

Successfully recreated pruned resources.

Step 1: Skip Pruning for a Resource

To ensure that a resource is not pruned, add the annotation harness.io/skipPruning: "true".

When Harness identifies resources from the last successful release which are not in current release, it searches for the harness.io/skipPruning annotation and ignores any resources that have it set to true.

You can deploy a resource using the annotation harness.io/skipPruning: "true", and then if the manifest is removed and another deployment occurs, Harness will see the annotation harness.io/skipPruning: "true" on the resource previously deployed and skip pruning it.

As mentioned in Limitations above, you cannot add a resource with the annotation outside of a Harness deployment and have Harness skip the pruning of that resource.

See Also

Configure As Code

To see how to configure the settings in this topic using YAML, configure the settings in the UI first, and then click the YAML editor button (</>).


How did we do?