I am currently installing istio 1.14.1 on a google kubernetes cluster (GKE), I am making the following manifest file:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
components:
base:
enabled: true
cni:
enabled: true
namespace: kube-system
values:
cni:
cniBinDir: /home/kubernetes/bin
cniConfDir: /etc/cni/net.d
chained: false
cniConfFileName: istio-cni.conf
meshConfig:
enableAutoMtls: true
I would like to know if it is well done and if someone can give me an orientation on how it should be configured correctly
Related
I am setting up istio controlplane using istio-operator on an EKS cluster with calico CNI. After installing istio on the cluster, I got to know that new pods are not coming up and the reason I got after googling is given below:
Istio Installation successful but not able to deploy POD
Now, I want to apply a change hostNetwork: true under spec.template.spec to istiod deployment using the istio-operator only.
I did some more googling to change or override the values of istiod deployment and got the following yamls files:
https://github.com/istio/istio/tree/ca541df418d0902ebeb9506c84d24c6bd9743801/operator/cmd/mesh/testdata/manifest-generate/input
But they are also not working. Below is the last configurations I have applied:
kind: IstioOperator
metadata:
namespace: istio-system
name: zeta-zone-istiocontrolplane
spec:
profile: minimal
values:
pilot:
resources:
requests:
cpu: 222m
memory: 333Mi
hostNetwork: true
unvalidatedValues:
hostNetwork: true
Can anybody help me to add hostNetwork: true under spec.template.spec to istiod deployment using the istio-operator only?
I was able to achieve that using the following YAML after a lot of hit and trials and checking logs of istio-operator:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: istiocontrolplane
spec:
profile: minimal
hub: docker.io/istio
tag: 1.10.3
meshConfig:
rootNamespace: istio-system
components:
base:
enabled: true
pilot:
enabled: true
namespace: istio-system
k8s:
overlays:
- kind: Deployment
name: istiod
patches:
- path: spec.template.spec.hostNetwork
value: true # OVERRIDDEN
I'm new in Istio.
My question is how can I detect failures in services that are already running in istio?
and if there is a failure, how to define particular percentage of traffic to a new version of a service?
thanks.
I recommend using Kiali. Kiali helps you understand the structure and health of your service mesh by monitoring traffic flow and report.
Kiali is a management console for an Istio-based service mesh. It provides dashboards, observability, and lets you operate your mesh with robust configuration and validation capabilities. It shows the structure of your service mesh by inferring traffic topology and displays the health of your mesh. Kiali provides detailed metrics, powerful validation, Grafana access, and strong integration for distributed tracing with Jaeger.
Detailed documentation for installing Kiali can be found in the Installation Guide.
I have created a simple example to demonstrate how useful Kiali is.
First, I created a db-app application with two available versions (v1 and v2) and exposed it using single service:
# cat db-app.yml
apiVersion: v1
kind: Service
metadata:
labels:
app: db-app
name: db-app
namespace: default
spec:
ipFamilies:
- IPv4
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: db-app
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: db-app
version: v1
name: db-app-v1
spec:
replicas: 1
selector:
matchLabels:
app: db-app
version: v1
template:
metadata:
labels:
app: db-app
version: v1
spec:
containers:
- image: nginx
name: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: db-app
version: v2
name: db-app-v2
spec:
replicas: 1
selector:
matchLabels:
app: db-app
version: v2
template:
metadata:
labels:
app: db-app
version: v2
spec:
containers:
- image: nginx
name: nginx
# kubectl apply -f db-app.yml
service/db-app created
deployment.apps/db-app-v1 created
deployment.apps/db-app-v2 created
# kubectl get pod,svc
NAME READY STATUS RESTARTS AGE
pod/db-app-v1-59c8fb999c-bs47s 2/2 Running 0 39s
pod/db-app-v2-56dbf4c8d6-q24vm 2/2 Running 0 39s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/db-app ClusterIP 10.102.36.142 <none> 80/TCP 39s
Additionally, to illustrate how we can split the traffic, I generated some traffic to the db-app application:
# kubectl run test-traffic --image=nginx
pod/test-traffic created
# kubectl exec -it test-traffic -- bash
root#test-traffic:/# for i in $(seq 1 100000); do curl 10.102.36.142; done
...
Now in Kiali UI in the Graph section we can see traffic flow:
In the Services section, we can easily split traffic between the v1 and v2 versions using the Traffic Shifting Wizard:
NOTE: Detailed tutorial can be found in the Kiali Traffic Shifting tutorial.
We can also monitor the status of our application. As an example, I broke the v1 version:
# kubectl set image deployment/db-app-v1 nginx=nnnginx
deployment.apps/db-app-v1 image updated
In Kiali UI we see errors in the v1 version:
I suggest you read the Kali Official Tutorial to learn the full capabilities of Kali.
I am bit new to istio and still learning. I have a use-case in which Istio is already deployed in istio-system namespace but I need to deploy istio ingress-gateway Pod in test-ns namespace using istioOperator. I am using istio 1.6.7.
From Istio docs, its mentioned to run this cmd:
istioctl manifest apply --set profile=default --filename=istio-ingress-values.yaml but this will create istiod Pods in istio-system which i donot want since its already created.
So, I ran below cmds to just create Ingress Gateway POD but can;t see any Pods or services created in test-ns. Kindly help if this is possible
kubectl apply -f istio-ingress-values.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: test-ns
name: testoperator
ingressGateways:
- enabled: true
name: istio-ingressgateway
namespace: test-ns
k8s:
env:
- name: ISTIO_META_ROUTER_MODE
value: sni-dnat
hpaSpec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: istio-ingressgateway
resources: {}
service:
ports:
- name: http2
port: 80
targetPort: 80
- name: https
port: 443
targetPort: 443
In Istio it is possible to tune configuration profiles.
As I can see, you are using the default profile, so I will describe how you can tune this configuration profile to create istio-ingressgateway in the test-ns namespace.
We can display the default profile settings by running the istioctl profile dump default command.
First, I saved these default settings in the default_profile_dump.yml file:
# istioctl profile dump default > default_profile_dump.yml
And then I modified this file:
NOTE: I only added one line: namespace: test-ns.
...
ingressGateways:
- enabled: true
name: istio-ingressgateway
namespace: test-ns
...
After modifying default settings of the ingressGateways, I applied these new settings:
# istioctl manifest apply -f default_profile_dump.yml
This will install the Istio 1.9.1 default profile with ["Istio core" "Istiod" "Ingress gateways"] components into the cluster. Proceed? (y/N) y
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
- Pruning removed resources Removed HorizontalPodAutoscaler:istio-system:istio-ingressgateway.
Removed PodDisruptionBudget:istio-system:istio-ingressgateway.
Removed Deployment:istio-system:istio-ingressgateway.
Removed Service:istio-system:istio-ingressgateway.
Removed ServiceAccount:istio-system:istio-ingressgateway-service-account.
Removed RoleBinding:istio-system:istio-ingressgateway-sds.
Removed Role:istio-system:istio-ingressgateway-sds.
✔ Installation complete
Finally, we can check where istio-ingressgateway was deployed:
# kubectl get pod -A | grep ingressgateway
test-ns istio-ingressgateway-7fc7c7c-r92tw 1/1 Running 0 33s
The istiod Deployment remained intact in the istio-system namespace:
# kubectl get deploy,pods -n istio-system
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/istiod 1/1 1 1 51m
NAME READY STATUS RESTARTS AGE
pod/istiod-64675984c5-xl97n 1/1 Running 0 51m
My setup: Running EKS 1.14 with Calico and Istio 1.5.
Trying to get the sample bookinfo to run with specific NetworkPolicies.
I've applied a GlobalNetworkPolicy that denies all traffic:
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: default-deny
spec:
selector: all()
types:
- Ingress
- Egress
I also added a GlobalNetworkPolicy for Istio to namespace and intra-namespace traffic:
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: allow-istio-system-to-ns
spec:
selector: all()
namespaceSelector: istio-injection == 'enabled'
types:
- Ingress
- Egress
ingress:
- action: Allow
source:
selector: all()
namespaceSelector: projectcalico.org/namespace == 'istio-system'
- action: Allow
source:
selector: all()
namespaceSelector: istio-injection == 'enabled'
egress:
- action: Allow
and a NetworkPolicy allowing all ingress and egress on istio-system
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
name: allow-istio-system-all
namespace: istio-system
spec:
selector: all()
types:
- Ingress
- Egress
ingress:
- action: Allow
egress:
- action: Allow
And a NetworkPolicy to allow traffic to the ingress-gateway on ports 80 and 443. I know this one is redundant, but I was hoping to cut down the previous one to only necessary ingress.
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
name: allow-istio-ingress-on-80-443
namespace: istio-system
spec:
selector: app == 'istio-ingressgateway'
ingress:
- action: Allow
protocol: TCP
destination:
ports:
- 80
- 443
Also some other, unrelated NetworkPolicies (access to kube-dns, metrics-server).
Deploying bookinfo works with and without the policies and same with deploying the Gateway.
The connection between the components works (I can exec into one pod and connect to the others).
But when I try to access the productpage via the gateway with the policies I get nothing back, instead of the HTML I get without the policies.
Am I missing some traffic that should be allowed? Should I have policies for the master nodes or for the loadbalancer somewhere else?
Edit: If I allow all ingress into istio-system and into the namespace, it works. So I think I'm just missing some other ingress from the namespace, is there a way to limit it to just the loadbalancer?
First of all there is a typo in Your allow-istio-system-to-ns yaml:
namespaceSelector: projectcalico.org/namespace == 'istio-system
There should be another ' at the end of the line.
Secondly this could be caused by the changes to policy and mixer in istio version 1.5.
According to Istio documentation:
The mixer policy is deprecated in Istio 1.5 and not recommended for production usage.
Rate limiting: Consider using Envoy native rate limiting instead of mixer rate limiting. Istio will add support for native rate limiting API through the Istio extensions API.
Control headers and routing: Consider using Envoy ext_authz filter, lua filter, or write a filter using the Envoy-wasm sandbox.
Denials and White/Black Listing: Please use the Authorization Policy for enforcing access control to a workload.
There is a guide in istio documentation which allows to turn back on depreciated features:
For an existing Istio mesh
Check the status of policy enforcement for your mesh.
$ kubectl -n istio-system get cm istio -o jsonpath="{#.data.mesh}" | grep disablePolicyChecks
disablePolicyChecks: true
If policy enforcement is enabled (disablePolicyChecks is false), no further action is needed.
Update the istio configuration to enable policy checks.
Execute the following command from the root Istio directory:
$ istioctl manifest apply --set values.global.disablePolicyChecks=false --set values.pilot.policy.enabled=true
configuration "istio" replaced
Validate that policy enforcement is now enabled.
$ kubectl -n istio-system get cm istio -o jsonpath="{#.data.mesh}" | grep disablePolicyChecks
disablePolicyChecks: false
Note that on calico documentation it says it requires the following istio versions:
Istio v1.0, v1.1, v1.2, or v1.3
Hope it helps.
For some reason it seems to work when I replace projectcalico.org/namespace == 'istio-system' with a label from istio-system (e.g. istio-injection=disabled).
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: allow-istio-system-to-ns
spec:
selector: all()
namespaceSelector: istio-injection == 'enabled'
types:
- Ingress
- Egress
ingress:
- action: Allow
source:
selector: all()
namespaceSelector: istio-injection == 'disabled'
- action: Allow
source:
selector: all()
namespaceSelector: istio-injection == 'enabled'
egress:
- action: Allow
I'm creating a new EKS Kubernetes Cluster on AWS.
When I deploy my workloads (migrating from an existing cluster) Kubelet stopps posting node status and all worker nodes become "NotReady" within a minute.
I was assuming that a misconfiguration within my cluster should not make the nodes crash - but apperently it does.
Can a misconfiguration within my cluster really make the AWS EKS Worker Nodes "NotReady"? Are there some rules of thumb under what circumstances this can happen? CPU Load to high? Pods in kube-system crashing?
This is a community wiki answer based on the solution from comments and posted for better visibility. Feel free to expand it.
As suggested by #gusto2 the problem was with the kubelet pod that was unable to call the API server. #stackoverflowjakob late confirmed that the connection between worker and master node was broken due to VPC misconfiguration and it was discovered by checking AWS Console -> EKS status.
Did you change the default PSP (pod security policy)? In my case, I added new eks.restricted psp, and new nodes will be NotReady status. My solution is to restore eks.privileged psp.
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: eks.privileged
annotations:
kubernetes.io/description: 'privileged allows full unrestricted access to
pod features, as if the PodSecurityPolicy controller was not enabled.'
seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
labels:
kubernetes.io/cluster-service: "true"
eks.amazonaws.com/component: pod-security-policy
spec:
privileged: true
allowPrivilegeEscalation: true
allowedCapabilities:
- '*'
volumes:
- '*'
hostNetwork: true
hostPorts:
- min: 0
max: 65535
hostIPC: true
hostPID: true
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'
readOnlyRootFilesystem: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: eks:podsecuritypolicy:privileged
labels:
kubernetes.io/cluster-service: "true"
eks.amazonaws.com/component: pod-security-policy
rules:
- apiGroups:
- policy
resourceNames:
- eks.privileged
resources:
- podsecuritypolicies
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: eks:podsecuritypolicy:authenticated
annotations:
kubernetes.io/description: 'Allow all authenticated users to create privileged pods.'
labels:
kubernetes.io/cluster-service: "true"
eks.amazonaws.com/component: pod-security-policy
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: eks:podsecuritypolicy:privileged
subjects:
- kind: Group
apiGroup: rbac.authorization.k8s.io
name: system:authenticated
You can try kubectl describe node $BAD_NODE or ssh into node and try sudo dmesg -T and
try restating the kubelet on the node /etc/init.d/kubelet restart
Or
systemctl restart kubelet
Or delete node (drain first)
kubectl drain <node-name>
kubectl delete node <node-name>