TKGs : Force vNamespace deletion

For months, I spent a lot of my efforts in the cloud-native world.
I set up in our lab environment vSphere with Tanzu and NSX ALB where I play with various cloud-native projects like ArgoCD, Harbor,…

I wanted to delete the namespace when… the worse happens:

I can see the namespace is in “terminating” status:

Then I’ve decided to go into the context and check what’s happening inside

Nothing was really helpful in the official documentation regarding my specific issue.

I tried my chance with the #CloudNative channel on vExpert Slack.
Few guys give me answers which I’ll try right now.

First, let’s start a proxy to communicate with the Kubernetes API:

kubectl proxy

And run the below command. The goal here is to edit a specific section in our YAML declaration for the namespace :

kubectl get namespace <YOUR-NAMESPACE> -o json | jq '.spec.finalizers=[]' | curl -X PUT http://localhost:8001/api/v1/namespaces/<YOUR-NAMESPACE>/finalize -H "Content-Type: application/json" --data @-

If I run the command from my workstation and logged as administrator@vsphere.local, It fails with a permission issue.
Then, let’s try to connect as root on the supervisor cluster.

First, retrieve the root password.
Connect as root on vCenter and type the below command:

/usr/lib/vmware-wcp/decryptK8Pwd.py

Type the same command from the supervisor machine as root and here is the output:

root@423d8ae0ef1a32190017422e40c04cf2 [ ~ ]# kubectl get namespace argo-cd -o json | jq '.spec.finalizers=[]' | curl -X PUT http://localhost:8001/api/v1/namespaces/argo-cd/finalize -H "Content-Type: application/json" --data @-
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "argo-cd",
    "selfLink": "/api/v1/namespaces/argo-cd/finalize",
    "uid": "0c9e90fe-7dc1-48a1-b1e9-65e7573022ee",
    "resourceVersion": "53061008",
    "creationTimestamp": "2021-08-18T06:44:59Z",
    "deletionTimestamp": "2021-08-23T07:09:51Z",
    "labels": {
      "vSphereClusterID": "domain-c8"
    },
    "annotations": {
      "vmware-system-resource-pool": "resgroup-15428",
      "vmware-system-vm-folder": "group-v15429"
    },
    "managedFields": [
      {
        "manager": "wcpsvc",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2021-08-18T06:45:10Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {"f:metadata":{"f:annotations":{".":{},"f:vmware-system-resource-pool":{},"f:vmware-system-vm-folder":{}},"f:labels":{".":{},"f:vSphereClusterID":{}}},"f:status":{"f:phase":{}}}
      },
      {
        "manager": "kube-controller-manager",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2021-08-23T07:10:04Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {"f:status":{"f:conditions":{".":{},"k:{\"type\":\"NamespaceContentRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionContentFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceFinalizersRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}
      }
    ]
  },
  "spec": {

  },
  "status": {
    "phase": "Terminating",
    "conditions": [
      {
        "type": "NamespaceDeletionDiscoveryFailure",
        "status": "False",
        "lastTransitionTime": "2021-08-23T07:09:58Z",
        "reason": "ResourcesDiscovered",
        "message": "All resources successfully discovered"
      },
      {
        "type": "NamespaceDeletionGroupVersionParsingFailure",
        "status": "False",
        "lastTransitionTime": "2021-08-23T07:09:58Z",
        "reason": "ParsedGroupVersions",
        "message": "All legacy kube types successfully parsed"
      },
      {
        "type": "NamespaceDeletionContentFailure",
        "status": "False",
        "lastTransitionTime": "2021-08-23T07:09:58Z",
        "reason": "ContentDeleted",
        "message": "All content successfully deleted, may be waiting on finalization"
      },
      {
        "type": "NamespaceContentRemaining",
        "status": "True",
        "lastTransitionTime": "2021-08-23T07:09:58Z",
        "reason": "SomeResourcesRemain",
        "message": "Some resources are remaining: clusters.cluster.x-k8s.io has 1 resource instances, kubeadmcontrolplanes.controlplane.cluster.x-k8s.io has 1 resource instances, machines.cluster.x-k8s.io has 2 resource instances, tanzukubernetesclusters.run.tanzu.vmware.com has 1 resource instances, virtualmachines.vmoperator.vmware.com has 2 resource instances, virtualmachinesetresourcepolicies.vmoperator.vmware.com has 1 resource instances, wcpmachines.infrastructure.cluster.vmware.com has 2 resource instances"
      },
      {
        "type": "NamespaceFinalizersRemaining",
        "status": "True",
        "lastTransitionTime": "2021-08-23T07:09:58Z",
        "reason": "SomeFinalizersRemain",
        "message": "Some content in the namespace has finalizers remaining: cluster.cluster.x-k8s.io in 1 resource instances, kubeadm.controlplane.cluster.x-k8s.io in 1 resource instances, machine.cluster.x-k8s.io in 2 resource instances, tanzukubernetescluster.run.tanzu.vmware.com in 1 resource instances, virtualmachine.vmoperator.vmware.com in 2 resource instances, virtualmachinesetresourcepolicy.vmoperator.vmware.com in 1 resource instances, wcpmachine.infrastructure.cluster.vmware.com in 2 resource instances"
      }
    ]
  }
}

And BOOM the vNamespace has gone, only my supervisor cluster is still here!!!

Feel free to comment on this article or on social media. Maybe, there is shorten path to correct this?