Excessive portal-sync-backup Pods in Pending Status Impacting Healthcheck

Step 1: Remove Excess Sync-Backup Jobs

for name in $(kubectl get job -o name | grep sync-backup); do
  kubectl delete $name
done

Verify that the jobs have been removed:

kubectl get job -o name | grep sync-backup | wc -l

Check if the pending pods have cleared:

kubectl get pods | grep sync-backup

Wait approximately 5 minutes to allow Kubernetes to clean up the pods.

Step 2: Restart Container Runtime (if pods persist)

If pending pods remain after job deletion, the container runtime (containerd) may be in a degraded state. To resolve :

systemctl status containerd
systemctl restart containerd
systemctl status containerd
kubectl get pods | grep sync-backup

Step 3: Check for Resource Exhaustion

If the issue persists and the following command fails:

crictl pods

With an error similar to:

code = ResourceExhausted desc = grpc: trying to send message larger than max (16835417 vs. 16777216)

Refer to Technote click here for instructions on resolving containerd gRPC message size limits.