We have an application running on EKS 1.21 and we are in the process of upgrading to EKS 1.22 since 1.21 will reach end of support in Feb 2023. I am trying to figure out a kubectl command or something similar that can query the entire cluster for any v1beta1 API versions that are deprecated and replaced by GA v1 as per :-
https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/
https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html#update-1.22
https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-22
Is there a comprehensive consolidated kubectl command that can be used to see if there is any YAML manifest in the cluster that still uses v1beta1?
The below command would return the namespace, name, apiVersion, and all the resources in your cluster. You can pipe this with grep to filter wherever you want.
kubectl api-resources --no-headers |while read type ignore; do kubectl get $type -A -o go-template='{{range $index,$pod := .items}}{{(or .metadata.name "-")}} {{(or .metadata.namespace "-")}} {{ (or .kind "-")}} {{ (or .apiVersion "-") }} {{"\n"}}{{end}}' 2>/dev/null; done
Related
I am running Kubeflow pipeline on a single node Rancher K3S cluster. Katib is deployed to create training jobs (Kind: TFJob) along with experiments (a CRD).
I can list the experiment resources with kubectl get experiments -n <namespace>. However, when trying to delete using kubectl delete experiment exp_name -n namespace the API server returns NotFound.
kubectl version is 1.22.12
kubeflow 1.6
How can a(any) resource be deleted when it is listed by "kubectl get , but a direct kubectl delete says the resource cannot be found?
Hopefully there is a general answer applicable for any resource.
Example:
kc get experiments -n <namespace>
NAME TYPE STATUS AGE
mnist-e2e Running True 21h
kc delete experiment mnist-e2e -n namespace
Error from server (NotFound): experiments.kubeflow.org "mnist-e2e" not found
I have tried these methods, but all involve the use of the resource name (mnist-e2e) and result in "NotFound".
I tried patching the manifest to empty the finalizers list:
kubectl patch experiment mnist-e2e \
-n namespace \
-p '{"metadata":{"finalizers":[]}}' \
--type=merge
I tried dumping a manifest of the "orphaned" resource and then deleting using that manifest:
kubectl get experiment mnist-e2e -n namespace -o yaml > exp.yaml
kubectl delete -f exp.yaml
Delete attempts from the Kubeflow UI Experiments (AutoML) page fail.
Thanks
I'm getting an error to migrate API from GKE though I'm not using the said API /apis/extensions/v1beta1/ingresses
I ran the command kubectl get deployment [mydeployment] -o yaml and did not find the API in question
It seems an IngressList is that calls the old API. To check you can use following command, this will give you the entire ingress info.
kubectl get --raw /apis/extensions/v1beta1/ingresses | jq
I have same issue but i have upgraded node version from 1.21 to 1.22
Istio question, where is pilot-discovery command?
i can found. In istio-1.8.0 directory has no command named pilot-discovery.
pilot-discovery command is command used by pilot, which is part of istiod now.
istiod unifies functionality that Pilot, Galley, Citadel and the sidecar injector previously performed, into a single binary.
You can get your istio pods with
kubectl get pods -n istio-system
Use kubectl exec to get into your istiod container with
kubectl exec -ti <istiod-pod-name> -c discovery -n istio-system -- /bin/bash
Use pilot-discovery commands as mentioned in istio documentation.
e.g.
istio-proxy#istiod-f49cbf7c7-fn5fb:/$ pilot-discovery version
version.BuildInfo{Version:"1.8.0", GitRevision:"c87a4c874df27e37a3e6c25fa3d1ef6279685d23", GolangVersion:"go1.15.5", BuildStatus:"Clean", GitTag:"1.8.0-rc.1"}
In case you are interested in the code: https://github.com/istio/istio/blob/release-1.8/pilot/cmd/pilot-discovery/main.go
I compile the binary by myself.
1 download istio project.
2 make build
3 set golang proxy
4 cd out
You will see the binary.
I am trying to upgrade my kubernetes version from 1.14 to 1.15, the cluster upgrade went well but when i am trying to update the node i am seeing the message
Couldn't proceed with upgrade process as new nodes are not joining node group standard-workers . I had created the nodes using eksctl.
I see the following error when i check the new node details under workloads.
runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
I also checked the tags associated with the new nodes getting spun up, they are all the same with one difference, the existing nodes have aws:ec2launchtemplate:version set to 1 and new nodes have it set to 4
I have checked the CNI plugin version and it corresponds to the latest once recommened
kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2
amazon-k8s-cni-init:v1.7.5-eksbuild.1
amazon-k8s-cni:v1.7.5-eksbuild.1
Any help on how to get around this would be really helpfull.
TIA
for others visiting the post. I found out that the issue was the url i was using to configure the kube proxy.
As per the documentation https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html, it says
Update kube-proxy to the recommended version by taking the output from the
previous step and replacing the version tag with your cluster's recommended
kube-proxy version:
kubectl set image daemonset.apps/kube-proxy \
-n kube-system \
kube-proxy=<602401143452.dkr.ecr.us-west-2.amazonaws.com>/eks/kube-proxy:v<1.18.9>-eksbuild.1
Your account ID and Region may differ from the example above.
I misunderstood the last part of account id and substituted it with my account id, which resulted in the image not getting found.
After using the correct link with account id 602401143452, i was able to fix the issue and node group upgrade was successful.
Hope this helps, Thanks.
You might have to update you CNI version. Please follow the CNI upgrade tutorial from here:
https://docs.aws.amazon.com/eks/latest/userguide/cni-upgrades.html
I'm running emr-5.12.0, with Amazon 2.8.3, Hive 2.3.2, Hue 4.1.0, Livy 0.4.0, Spark 2.2.1 and Zeppelin 0.7.3 on 1 m4.large as my master node and 1 m4.large as core node.
I am trying to execute a bootstrap action that configures some parts of the cluster. One of these includes the line:
sudo sed -i '/zeppelin.pyspark.python/c\ \"zepplin.pyspark.python\" : \"python3\",' /etc/alternatives/zeppelin-conf/interpreter.json
It makes sure that the Zeppelin uses python3.4 instead of python2.7. It works fine if I execute this in the terminal after SSH'ing to the master node, but it fails when I submit it as a Custom JAR step on the AWS Web interface. I get the following error:
ed: can't read /etc/alternatives/zeppelin-conf/interpreter.json
: No such file or directory
Command exiting with ret '2'
The same thing happens if I use
sudo sed -i '/zeppelin.pyspark.python/c\ \"zepplin.pyspark.python\" : \"python3\",' /etc/zeppelin-conf/interpreter.json
Obviously I could just change it from the Zeppelin UI, but I would like to include it in the bootstrap action.
Thanks!
It turns out that a bootstrap action submitted throug the AWS EMR web interface is submitted as a regular EMR step, so it's only run on the master node. This can be seen if you click the 'AWS CLI export' in the cluster web interface. The intended bootstrap action is listed as a regular step.
Using the command line to launch a cluster with a bootstrap action bypasses this problem, so I've just used that.
Edit: Looking back at the web interface, it's pretty clear that I was adding regular steps instead of bootstrap actions. My bad!