We have Istio set up and running in our clusters, with automatic injection enabled by default and enabled in a handful of namespaces. Now we want to do automatic injection for some pods in some other namespaces, but encountered a problem that it is seemingly impossible to do an automatic injection for a specified pod if it is not enabled for the whole namespace. We use Argo workflows to create pods automatically, so we specify sidecar.istio.io/inject: "true" inside Argo workflows so that the resulting pods appear with this annotation in their metadata:
...
metadata:
annotations:
sidecar.istio.io/inject: "true"
...
Unfortunately, Istio still does not inject a sidecar unless the namespace has the istio-injection label explicitly set to enabled, adding sidecars to all pods running there.
We cannot use the manual injection either since the pods are created automatically by the Argo service, and we wanted the sidecars to be injected only to specific pods based on the workflow definition.
So are there any possible ways to overcome this issue? Thanks!
Full Argo workflow:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: presto-sql-pipeline-
annotations: {pipelines.kubeflow.org/kfp_sdk_version: 0.5.1, pipelines.kubeflow.org/pipeline_compilation_time: '2020-05-16T16:07:29.173967',
pipelines.kubeflow.org/pipeline_spec: '{"description": "Simple demo of Presto
SQL operator PrestoSQLOp", "name": "Presto SQL Pipeline"}'}
labels: {pipelines.kubeflow.org/kfp_sdk_version: 0.5.1}
spec:
entrypoint: presto-sql-pipeline
templates:
- name: presto-demo
container:
args:
- --source-name
- '{{workflow.namespace}}.{{workflow.name}}.presto-demo'
- --query-sql
- "SELECT 1;"
image: gcr.io/our-data-warehouse/presto-cli:latest
volumeMounts:
- {mountPath: /mnt/secrets, name: presto-local-vol}
metadata:
annotations:
sidecar.istio.io/inject: "true"
labels: {pipelines.kubeflow.org/pipeline-sdk-type: kfp}
volumes:
- name: presto-local-vol
secret: {secretName: presto-local}
- name: presto-sql-pipeline
dag:
tasks:
- {name: presto-demo, template: presto-demo}
arguments:
parameters: []
serviceAccountName: argo
I had a similar requirement - Istio should inject proxy only when specified by the pod and ignore auto injection for all other pods.
The solution isnt mentioned in the official docs of Istio but its possible to do so.
As given in this user defined custom matrix, we can have Istio follow this behaviour when the following conditions are met :
The namespace has the label istio-injection=enabled
The Istio global proxy auto inject policy is disabled (For helm chart value : global.proxy.autoInject as given here).
The pod which needs the proxy has the annotation sidecar.istio.io/inject: "true".
All other pods will not have the Istio proxy.
Related
I'm trying to create a Kubernetes deployment with an associated ServiceAccount, which is linked to an AWS IAM role. This yaml produces the desired result and the associated deployment (included at the bottom) spins up correctly:
apiVersion: v1
kind: ServiceAccount
metadata:
name: service-account
namespace: example
annotations:
eks.amazonaws.com/role-arn: ROLE_ARN
However, I would like to instead use the Terraform Kubernetes provider to create the ServiceAccount:
resource "kubernetes_service_account" "this" {
metadata {
name = "service-account2"
namespace = "example"
annotations = {
"eks.amazonaws.com/role-arn" = "ROLE_ARN"
}
}
}
Unfortunately, when I create the ServiceAccount this way, the ReplicaSet for my deployment fails with the error:
Error creating: Internal error occurred: Internal error occurred: jsonpatch add operation does not apply: doc is missing path: "/spec/volumes/0"
I have confirmed that it does not matter whether the Deployment is created via Terraform or kubectl; it will not work with the Terraform-created service-account2, but works fine with the kubectl-created service-account. Switching a deployment back and forth between service-account and service-account2 correspondingly makes it work or not work as you might expect.
I have also determined that the eks.amazonaws.com/role-arn is related; creating/assigning ServiceAccounts that do not try to link back to an IAM role work regardless of whether they were created via Terraform or kubectl.
Using kubectl to describe the Deployment, ReplicaSet, ServiceAccount, and associated Secret, I don't see any obvious differences, though I will admit I'm not entirely sure what I might be looking for.
Here is a simple deployment yaml that exhibits the problem:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: example
namespace: example
spec:
strategy:
type: Recreate
template:
metadata:
labels:
app: example
spec:
serviceAccountName: service-account # or "service-account2"
containers:
- name: nginx
image: nginx:1.7.8
Adding automountServiceAccountToken: true to the pod spec in your deployment should fix this error. This is usually enabled by default on service accounts, but Terraform defaults it to off. See this issue on the mutating web hook that adds the required environment variables to your pods: https://github.com/aws/amazon-eks-pod-identity-webhook/issues/17
I had the same problem, and I solved it specifying automount_service_account_token = true in the terraform kubernetes service account resource.
Try crating the following service account:
resource "kubernetes_service_account" "this" {
metadata {
name = "service-account2"
namespace = "example"
annotations = {
"eks.amazonaws.com/role-arn" = "ROLE_ARN"
}
}
automount_service_account_token = true
}
I'm trying out AWS EKS following this guide https://learn.hashicorp.com/terraform/aws/eks-intro
I understand all the steps except for the last one where the instructions say to apply a configmap. Before this step, I couldn't see my worker nodes from the cluster kubectl get nodes. But, I can see my worker nodes after applying this configmap. Can someone please explain to me how this configmap actually accomplishes this feat.
Here is the configmap:
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- rolearn: ${aws_iam_role.demo-node.arn}
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
Thanks,
Ashutosh Singh
The data in that configmap are what enables the worker node to join the cluster. Specifically, it needs the proper role ARN permissions. In the tutorial you are following, look at how aws_iam_role.demo-node.arn is defined, then look up the permissions associated with those policies. You can experiment around and remove/add other policies and see how it affects the node's ability to interact with the cluster.
I have a deploymgr template that creates a bunch of network assets and VMs and it runs fine with no errors reported, however no VPC peerings are ever created. It works fine if I create a peering via the console or on the cli via glcoud
Peering fails (with no error msg):
# Create the required routes to talk to prod project
- name: mytest-network
type: compute.v1.network
properties:
name: mytest
autoCreateSubnetworks: false
peerings:
- name: mytest-to-prod
network: projects/my-prod-project/global/networks/default
autoCreateRoutes: true
Peering Works:
$ gcloud compute networks peerings create mytest-to-prod --project=myproject --network=default --peer-network=projects/my-prod-project/global/networks/default --auto-create-routes
The Peering cannot be done at network creation time as per the API reference.
First the network needs to be created and once it has been created successfully, the addPeering method needs to be called.
This explains why your YAML definition created the network but not the peering and it worked after running the gcloud command that it calls the addPeering method.
There is a possibility of creating and doing the peering on one YAML file by using the Deployment manager actions:
resources:
- name: mytest-network1
type: compute.v1.network
properties:
name: mytest1
autoCreateSubnetworks: false
- name: mytest-network2
type: compute.v1.network
properties:
name: mytest2
autoCreateSubnetworks: false
- name: addPeering2-1
action: gcp-types/compute-v1:compute.networks.addPeering
metadata:
runtimePolicy:
- CREATE
properties:
network: mytest-network2
name: vpc-2-1
autoCreateRoutes: true
peerNetwork: $(ref.mytest-network1.selfLink)
metadata:
dependsOn:
- mytest-network1
- mytest-network2
- name: addPeering1-2
action: gcp-types/compute-v1:compute.networks.addPeering
metadata:
runtimePolicy:
- CREATE
properties:
network: mytest-network1
name: vpc-1-2
autoCreateRoutes: true
peerNetwork: $(ref.mytest-network2.selfLink)
metadata:
dependsOn:
- mytest-network1
- mytest-network2
You can copy-paste the YAML above, create the deployment and the peering should be done. The actions use the dependsOn option to make sure the network are created first and when deleting the deployment the peerings would be deleted by calling the removePeering method and then the networks would be deleted.
Note: The Deployment manager actions are undocumented yet but there are several examples in the GoogleCloudPlatform/deploymentmanager-samples repository such as this and this.
From gcloud works as expected, please update your YAML file to use "peerings[].network" when specifying the list of peered network resources.
I'm using a slightly customized Terraform configuration to generate my Kubernetes cluster on AWS. The configuration includes an EFS instance attached to the cluster nodes and master. In order for Kubernetes to use this EFS instance for volumes, my Kubernetes YAML needs the id and endpoint/domain of the EFS instance generated by Terraform.
Currently, my Terraform outputs the EFS id and DNS name, and I need to manually edit my Kubernetes YAML with these values after terraform apply and before I kubectl apply the YAML.
How can I automate passing these Terraform output values to Kubernetes?
I don't know what you mean by a yaml to set up an Kubernetes cluster in AWS. But then, I've always set up my AWS clusters using kops. Additionally I don't understand why you would want to mount an EFS to the master and/or nodes instead of to the containers.
But in direct answer to your question: you could write a script to output your Terraform outputs to a Helm values file and use that to generate the k8s config.
I stumbled upon this question when searching for a way to get TF outputs to envvars specified in Kubernetes and I expect more people do. I also suspect that that was really your question as well or at least that it can be a way to solve your problem. So:
You can use the Kubernetes Terraform provider to connect to your cluster and then use the kubernetes_config_map resources to create configmaps.
provider "kubernetes" {}
resource "kubernetes_config_map" "efs_configmap" {
"metadata" {
name = "efs_config" // this will be the name of your configmap
}
data {
efs_id = "${aws_efs_mount_target.efs_mt.0.id}"
efs_dns = "${aws_efs_mount_target.efs_mt.0.dns_name}"
}
}
If you have secret parameters use the kubernetes_secret resource:
resource "kubernetes_secret" "some_secrets" {
"metadata" {
name = "some_secrets"
}
data {
s3_iam_access_secret = "${aws_iam_access_key.someresourcename.secret}"
rds_password = "${aws_db_instance.someresourcename.password}"
}
}
You can then consume these in your k8s yaml when setting your environment:
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: some-app-deployment
spec:
selector:
matchLabels:
app: some
template:
metadata:
labels:
app: some
spec:
containers:
- name: some-app-container
image: some-app-image
env:
- name: EFS_ID
valueFrom:
configMapKeyRef:
name: efs_config
key: efs_id
- name: RDS_PASSWORD
valueFrom:
secretKeyRef:
name: some_secrets
key: rds_password
I have an OpenShift deployment configuration template that I generated from a working deployment (using "oc export"). The original pod has a persistent volume claim (PVC) mounted on /data. When I try to deploy using the template, the pod never starts up. If I remove all mention of the volume and volume mount from the template, the pod does start. I then have to manually attach the volume. I want to be able to do it all from the template though. Here is the partial template showing only relevant items:
apiVersion: v1
kind: Template
metadata:
name: myapp
objects:
- apiVersion: v1
kind: DeploymentConfig
metadata:
name: myapp-service
spec:
template:
spec:
containers:
- name: myapp-service
image: my-private-registry/myapp-service:latest
volumeMounts:
- mountPath: /data
name: volume-001
volumes:
- persistentVolumeClaim:
claimName: nfs-pvc
name: volume-001
When deployed with this template, the deployment sits waiting for the pod to be created ("Status: Container creating"). If the persistentVolumeClaim item is replaced with an ephemeral volume declaration:
volumes:
- emptyDir: {}
name: volume-001
it starts up fine. So it seems that the problem is specific to the persistentVolumeClaim entry. Both the PV and PVC were set up beforehand, as shown here:
> oc get pv
NAME CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
nfs-pv00 50Gi RWX Bound my-project/nfs-pvc 1h
> oc get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
nfs-pvc Bound nfs-pv00 50Gi RWX 1h
EDIT: Another data point (that perhaps I should have started with) is that if an existing deployment configuration is modified from the openshift console (Applications->Deployments->my-dc) by selecting "Attach storage" and specifying the PVC, the same behavior is observed as with the templated deployment: a new deployment launches (due to config change) but its associated pod never starts.
SOLVED: My bad. I discovered that I had a bad mount point on the NFS server that I had set up to serve the PV. Once I specified the correct mount point, the pods started up fine (well, on to the next problem anyway).