Delete attempt of Kubernetes resource reports not found, even though it can be listed with "kubectl get" - kubectl

I am running Kubeflow pipeline on a single node Rancher K3S cluster. Katib is deployed to create training jobs (Kind: TFJob) along with experiments (a CRD).
I can list the experiment resources with kubectl get experiments -n <namespace>. However, when trying to delete using kubectl delete experiment exp_name -n namespace the API server returns NotFound.
kubectl version is 1.22.12
kubeflow 1.6
How can a(any) resource be deleted when it is listed by "kubectl get , but a direct kubectl delete says the resource cannot be found?
Hopefully there is a general answer applicable for any resource.
Example:
kc get experiments -n <namespace>
NAME TYPE STATUS AGE
mnist-e2e Running True 21h
kc delete experiment mnist-e2e -n namespace
Error from server (NotFound): experiments.kubeflow.org "mnist-e2e" not found
I have tried these methods, but all involve the use of the resource name (mnist-e2e) and result in "NotFound".
I tried patching the manifest to empty the finalizers list:
kubectl patch experiment mnist-e2e \
-n namespace \
-p '{"metadata":{"finalizers":[]}}' \
--type=merge
I tried dumping a manifest of the "orphaned" resource and then deleting using that manifest:
kubectl get experiment mnist-e2e -n namespace -o yaml > exp.yaml
kubectl delete -f exp.yaml
Delete attempts from the Kubeflow UI Experiments (AutoML) page fail.
Thanks

Related

Command to query list of all objects/resources using a specific API version in kubernetes

We have an application running on EKS 1.21 and we are in the process of upgrading to EKS 1.22 since 1.21 will reach end of support in Feb 2023. I am trying to figure out a kubectl command or something similar that can query the entire cluster for any v1beta1 API versions that are deprecated and replaced by GA v1 as per :-
https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/
https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html#update-1.22
https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-22
Is there a comprehensive consolidated kubectl command that can be used to see if there is any YAML manifest in the cluster that still uses v1beta1?
The below command would return the namespace, name, apiVersion, and all the resources in your cluster. You can pipe this with grep to filter wherever you want.
kubectl api-resources --no-headers |while read type ignore; do kubectl get $type -A -o go-template='{{range $index,$pod := .items}}{{(or .metadata.name "-")}} {{(or .metadata.namespace "-")}} {{ (or .kind "-")}} {{ (or .apiVersion "-") }} {{"\n"}}{{end}}' 2>/dev/null; done

Migrate to updated APIs

I'm getting an error to migrate API from GKE though I'm not using the said API /apis/extensions/v1beta1/ingresses
I ran the command kubectl get deployment [mydeployment] -o yaml and did not find the API in question
It seems an IngressList is that calls the old API. To check you can use following command, this will give you the entire ingress info.
kubectl get --raw /apis/extensions/v1beta1/ingresses | jq
I have same issue but i have upgraded node version from 1.21 to 1.22

Istio question, where is pilot-discovery command?

Istio question, where is pilot-discovery command?
i can found. In istio-1.8.0 directory has no command named pilot-discovery.
pilot-discovery command is command used by pilot, which is part of istiod now.
istiod unifies functionality that Pilot, Galley, Citadel and the sidecar injector previously performed, into a single binary.
You can get your istio pods with
kubectl get pods -n istio-system
Use kubectl exec to get into your istiod container with
kubectl exec -ti <istiod-pod-name> -c discovery -n istio-system -- /bin/bash
Use pilot-discovery commands as mentioned in istio documentation.
e.g.
istio-proxy#istiod-f49cbf7c7-fn5fb:/$ pilot-discovery version
version.BuildInfo{Version:"1.8.0", GitRevision:"c87a4c874df27e37a3e6c25fa3d1ef6279685d23", GolangVersion:"go1.15.5", BuildStatus:"Clean", GitTag:"1.8.0-rc.1"}
In case you are interested in the code: https://github.com/istio/istio/blob/release-1.8/pilot/cmd/pilot-discovery/main.go
I compile the binary by myself.
1 download istio project.
2 make build
3 set golang proxy
4 cd out
You will see the binary.

Knative on GKE is not working with some images, shows RevisionMissing error

I am running Knative on a GKE cluster. The sample images provided on the Knative website work but when I switch to some other images, it stops working. Only 2 containers work out of 3 and route's ready state remains 'unknown' and Reason shows as 'RevisionMissing'.
I tried with multiple images, k8s.gcr.io/hpa-example is one of them.
Edit: The cluster has a two-node of configuration of type n1-standard-4 (4 vCPUs, 15 GB memory). I created this cluster using the GCP console with the latest version of kubernetes, and checking the Enable Istio checkbox. I used following commands to install the Knative:
kubectl apply --selector knative.dev/crd-install=true \
-f https://github.com/knative/serving/releases/download/v0.8.0/serving.yaml \
-f https://github.com/knative/eventing/releases/download/v0.8.0/release.yaml \
-f https://github.com/knative/serving/releases/download/v0.8.0/monitoring.yaml
kubectl apply \
-f https://github.com/knative/serving/releases/download/v0.8.0/serving.yaml \
-f https://github.com/knative/eventing/releases/download/v0.8.0/release.yaml \
-f https://github.com/knative/serving/releases/download/v0.8.0/monitoring.yaml
Thanks
Ok, I found the problem. I tried posting custom images. All worked until I change the port (inside image) to 80. This image not only work as Knative service, but also, It did not work on Cloud run service as well.
Bottom line is, either pull port number from environment variable, or hard code it to any other port than 80.
Thanks for the precisions.
When you installed Knative you should see this kind of errors
# Without CRD
unable to recognize "https://github.com/knative/serving/releases/download/v0.8.0/serving.yaml": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
unable to recognize "https://github.com/knative/serving/releases/download/v0.8.0/serving.yaml": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
unable to recognize "https://github.com/knative/serving/releases/download/v0.8.0/serving.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"
unable to recognize "https://github.com/knative/eventing/releases/download/v0.8.0/release.yaml": no matches for kind "ClusterChannelProvisioner" in version "eventing.knative.dev/v1alpha1"
# Without CRD
Error from server (NotFound): error when creating "https://github.com/knative/serving/releases/download/v0.8.0/monitoring.yaml": namespaces "istio-system" not found
Error from server (NotFound): error when creating "https://github.com/knative/serving/releases/download/v0.8.0/monitoring.yaml": namespaces "istio-system" not found
Error from server (NotFound): error when creating "https://github.com/knative/serving/releases/download/v0.8.0/monitoring.yaml": namespaces "istio-system" not found
Error from server (NotFound): error when creating "https://github.com/knative/serving/releases/download/v0.8.0/monitoring.yaml": namespaces "istio-system" not found
You didn't have installed Istio. Do it, relaunch the knative installation (with and without CRD) to solve previous errors and enjoy!!!

Kubectl : No resource found

I’ve installed ICP4Data successfully. I am pretty green in respect to ICP4Data and Kubernetes. I’m trying to use kubectl command for listing the pods in ICP4D but “kubectl get pods” returns “No resource found”. Am I missing something?
icp4d uses 'zen' namespaces to logically separate its assets and resources from the core native icp/kube platform. In the default installation of ICP4D, there are no pods deployed on 'default' namespace and hence you get "no resources found" cause if you don't provide the namespace while trying to get pods, kubectl assumes its default namespace.
To List the pods from zen namespace
kubectl get pods -n zen
To list all the namespaces available to you - try
kubectl get namespaces
To list pods from all the namespaces, you might want to append --all-namespaces
kubectl get pods --all-namespaces
This should list all the pods from zen, kubesystem and possibly others.
Please try adding namespace to the command as well. In the case for ICP4D try kubectl get pods -n zen.
On the other hand, you could switch your namespace to zen at the beginning by
kubectl config set-context --current --namespace=zen
Then you will be able to see all the information by running without the -n argument
kubectl get pods
Check you are currently on which namespace.
To find out your pod is created in which namespace, you can run this command
kubectl get pods --all-namespaces
Also just to add, since I was in default workspace and I wanted to get logs of a pod in another namespace, just doing
kubectl get logs -f <pod_name>
was giving output "Error from server (NotFound): pods "pod_name" not found".
So I specified the namespace as well.
kubectl logs -f <pod_name> -n namespace