My vault is deployed in EKS cluster with S3 backend but whenever I try to access UI in the browser it gives 404.
This is my values.yaml file
server:
affinity: null
dataStorage:
enabled: false
dev:
enabled: false
standalone:
enabled: false
ha:
enabled: true
replicas: 1
config: |
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "s3" {
access_key = "key"
secret_key = "key"
bucket = "name"
region = "region"
}
ui:
enabled: true
serviceType: "LoadBalancer"
HTTP status Code 404 means that 404 is returned if the specified resource such as a vault, upload ID, or job ID does not exist.
Make sure that you have properly handled Vault - vault-404.
Take a look: 404-http-code-amazon.
Don't you have extraSecretEnvironmentVars ? Please fill proper field of file with proper values.
Add following block of code under server standalone section:
service:
enabled: true
Look at: standalone-service-vault.
Your values.json file should be in YAML format. What is more important is that it's not even valid YAML, as it has a block of JSON plonked in the middle of it .
Look at the proper example here: vault-eks.
It is solved I just have to put external Port for service
enabled: true
serviceType: "LoadBalancer"
externalPort: 8200
Related
I have a GKE cluster on which i have deployed OAuth proxy to protect some urls :
resource "helm_release" "oauth_proxy" {
name = "oauth-proxy"
namespace = "kube-system"
repository = "https://oauth2-proxy.github.io/manifests"
chart = "oauth2-proxy"
version = "6.2.0"
values = [<<EOF
config:
clientID: 'myClientID'
clientSecret: 'myClientSecret'
cookieSecret: 'myCookieSecret'
configFile: |-
email_domains = [ "mywebsite.com" ]
upstreams = [ "file:///dev/null" ]
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: lets-encrypt-production
path: /oauth2
hosts:
- url1
tls:
- secretName: myapp-tls
hosts:
- url1
EOF
]
}
So my concern is about the ClientID, secret and cookie. I have to deploy this to other projects (that have not the same IDs) and i don't want the credentials to remain clear like this in the code.
I've found that there's a way to do so with the google_iap_client terraform resource but the documentation shows only the brand config. is this a good way ?
Can it be done through a secret ?
I have Cloud Gateway API that connect to Cloud Function.
It worked great with an API key that I generated, but then when I restricted the key to enable access for my specific domain only - example.com, I started to get CORS errors on the client. ( it did worked without domain restriction)
So I enabled CORS in the config file in the Console, the file had no errors, but I still get a CORS error when accessing the API URL from my domain with the key:
https://project-xxxx.uc.gateway.dev/search?key=AIxxxxxxxxx
Config :
swagger: "2.0"
info:
title: projectapi
description: "data api"
version: "1.0.0"
schemes:
- "https"
host: "project-xxxxxxxx.apigateway.xxxxx-xxxxx.cloud.goog"
x-google-endpoints:
- name: "project-xxxxxxxx.apigateway.xxxxx-xxxxx.cloud.goog"
allowCors: True
paths:
"/search":
get:
description: "data"
operationId: "project"
x-google-backend:
address: https://us-central1-projectName.cloudfunctions.net/search
security:
- api_key: [key]
responses:
200:
description: "Success."
schema:
type: string
400:
description: "Invalid"
securityDefinitions:
api_key:
type: "apiKey"
name: "key"
in: "query"
My Cloud Function do allow CORS, and works great if called directly with the link here in the address key.
Why this config wont allow cors for the Gateway ?
Solved !
You must enable the key not only for your domain name, but also for your specific API name from the list, go to APIs & Services => Credentials => API keys => your key => API restrictions => choose your API name from the list !
This allow access to a Function from a specific domain - without using a Balancer.
I currently have the following problem. I have a backend that is behind an nginx-ingress controller used as load balancer in aws. Usually i should get the users real ip by either the header x-forwarded-for or x-real-ip. However this ip always points to the ip of the ingress controller in my case. I use terraform to setup my architecture.
This is my current configuration
resource "helm_release" "ingress_nginx" {
name = "ingress-nginx"
namespace = "ingress"
create_namespace = true
chart = "ingress-nginx"
version = "4.0.13"
repository = "https://kubernetes.github.io/ingress-nginx"
values = [
<<-EOF
controller:
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
admissionWebhooks:
enabled: false
EOF
]
}
data "kubernetes_service" "ingress_nginx" {
metadata {
name = "ingress-nginx-controller"
namespace = helm_release.ingress_nginx.metadata[0].namespace
}
}
data "aws_lb" "ingress_nginx" {
name = regex(
"(^[^-]+)",
data.kubernetes_service.ingress_nginx.status[0].load_balancer[0].ingress[0].hostname
)[0]
}
output "lb" {
value = data.aws_lb.ingress_nginx.name
}`
Does anybody know how to fix the header in this config?
To get the real IP address in the header both the ingress controller and the NLB (Network Load Balancer) need to use Proxy Protocol. To do this, in terraform you need to configure the ingress controller with the proxy-protocol option:
resource "helm_release" "ingress_nginx" {
name = "ingress-nginx"
namespace = "ingress"
create_namespace = true
chart = "ingress-nginx"
version = "4.0.13"
repository = "https://kubernetes.github.io/ingress-nginx"
values = [
<<-EOF
controller:
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
config:
use-forwarded-headers: true
use-proxy-protocol: true
enable-real-ip: true
admissionWebhooks:
enabled: false
EOF
]
}
data "kubernetes_service" "ingress_nginx" {
metadata {
name = "ingress-nginx-controller"
namespace = helm_release.ingress_nginx.metadata[0].namespace
}
}
data "aws_lb" "ingress_nginx" {
name = regex(
"(^[^-]+)",
data.kubernetes_service.ingress_nginx.status[0].load_balancer[0].ingress[0].hostname
)[0]
}
output "lb" {
value = data.aws_lb.ingress_nginx.name
}
Now, it seems that currently there is no way to get the NLB to use proxy protocol by annotating the service.
There are some issues in github mentioning that the following annotations would solve it:
*service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip"*
and
*service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"*
However, these have not worked for me. So, I have enabled proxy protocol manually though the AWS Console by doing the following:
AWS console --> EC2 --> Target groups under load balancing --> Select Target groups associated with your NLB --> Attributes --> Enable proxy protocol 2
I guess that in terraform the optimal solution would be to explicitly create a Network Load Balancer with the correct target group and associate it with the ingress controller service.
Doing this, now I get the correct IP addresses of the clients connecting to the ingress controller.
Objective
My objective is to be able to deploy on AWS EKS using Fargate. I have successfully made the deployment work with a node_group. However, when I shifted to using Fargate, it seems that the pods are all stuck in the pending state.
How my current code looks like
I am provisioning using Terraform (not necessarily looking for a Terraform answer). This is how I create my EKS Cluster:
module "eks_cluster" {
source = "terraform-aws-modules/eks/aws"
version = "13.2.1"
cluster_name = "${var.project_name}-${var.env_name}"
cluster_version = var.cluster_version
vpc_id = var.vpc_id
cluster_enabled_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
enable_irsa = true
subnets = concat(var.private_subnet_ids, var.public_subnet_ids)
create_fargate_pod_execution_role = false
node_groups = {
my_nodes = {
desired_capacity = 1
max_capacity = 2
min_capacity = 1
instance_type = var.nodes_instance_type
subnets = var.private_subnet_ids
}
}
}
And this is how I provision the Fargate profile:
resource "aws_eks_fargate_profile" "airflow" {
cluster_name = module.eks_cluster.cluster_id
fargate_profile_name = "${var.project_name}-fargate-${var.env_name}"
pod_execution_role_arn = aws_iam_role.fargate_iam_role.arn
subnet_ids = var.private_subnet_ids
selector {
namespace = "airflow"
}
}
And this is how I created and attach the required policies:
resource "aws_iam_role" "fargate_iam_role" {
name = "${var.project_name}-fargate-${var.env_name}"
force_detach_policies = true
assume_role_policy = jsonencode({
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = ["eks-fargate-pods.amazonaws.com", "eks.amazonaws.com"]
}
}]
Version = "2012-10-17"
})
}
# Attach IAM Policy for Fargate
resource "aws_iam_role_policy_attachment" "fargate_pod_execution" {
role = aws_iam_role.fargate_iam_role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSFargatePodExecutionRolePolicy"
}
What I have tried and does not work
I have tried deploying the pods (I am using a Helm chart) in the same namespace where the Fargate Profile Exists. When I run kubectl get pods -n airflow I see all my pods pending like:
NAME READY STATUS RESTARTS AGE
airflow-flower-79b5948677-vww5d 0/1 Pending 0 40s
airflow-redis-master-0 0/1 Pending 0 40s
airflow-scheduler-6b6bd4b6f6-j9qzg 0/2 Pending 0 41s
airflow-web-567b55fbbf-z8dsg 0/2 Pending 0 41s
airflow-worker-0 0/2 Pending 0 40s
airflow-worker-1 0/2 Pending 0 40s
Then I look at the events by kubectl get events -n airflow, to which I get:
LAST SEEN TYPE REASON OBJECT MESSAGE
2m15s Normal LoggingEnabled pod/airflow-flower-79b5948677-vww5d Successfully enabled logging for pod
2m16s Normal SuccessfulCreate replicaset/airflow-flower-79b5948677 Created pod: airflow-flower-79b5948677-vww5d
2m17s Normal ScalingReplicaSet deployment/airflow-flower Scaled up replica set airflow-flower-79b5948677 to 1
2m15s Normal LoggingEnabled pod/airflow-redis-master-0 Successfully enabled logging for pod
2m16s Normal SuccessfulCreate statefulset/airflow-redis-master create Pod airflow-redis-master-0 in StatefulSet airflow-redis-master successful
2m15s Normal LoggingEnabled pod/airflow-scheduler-6b6bd4b6f6-j9qzg Successfully enabled logging for pod
2m16s Normal SuccessfulCreate replicaset/airflow-scheduler-6b6bd4b6f6 Created pod: airflow-scheduler-6b6bd4b6f6-j9qzg
2m17s Normal NoPods poddisruptionbudget/airflow-scheduler No matching pods found
2m17s Normal ScalingReplicaSet deployment/airflow-scheduler Scaled up replica set airflow-scheduler-6b6bd4b6f6 to 1
2m15s Normal LoggingEnabled pod/airflow-web-567b55fbbf-z8dsg Successfully enabled logging for pod
2m16s Normal SuccessfulCreate replicaset/airflow-web-567b55fbbf Created pod: airflow-web-567b55fbbf-z8dsg
2m17s Normal ScalingReplicaSet deployment/airflow-web Scaled up replica set airflow-web-567b55fbbf to 1
2m15s Normal LoggingEnabled pod/airflow-worker-0 Successfully enabled logging for pod
2m15s Normal LoggingEnabled pod/airflow-worker-1 Successfully enabled logging for pod
2m16s Normal SuccessfulCreate statefulset/airflow-worker create Pod airflow-worker-0 in StatefulSet airflow-worker successful
2m16s Normal SuccessfulCreate statefulset/airflow-worker create Pod airflow-worker-1 in StatefulSet airflow-worker successful
I then try to describe one of the pods (via kubectl describe pod), and I get:
Name: airflow-redis-master-0
Namespace: airflow
Priority: 2000001000
Priority Class Name: system-node-critical
Node: <none>
Labels: app=redis
chart=redis-10.5.7
controller-revision-hash=airflow-redis-master-588d57785d
eks.amazonaws.com/fargate-profile=airflow-fargate-airflow-dev
release=airflow
role=master
statefulset.kubernetes.io/pod-name=airflow-redis-master-0
Annotations: CapacityProvisioned: 0.25vCPU 0.5GB
Logging: LoggingEnabled
checksum/configmap: 2b82c78fd9186045e6e2b44cfbb38460310697cf2f2f175c9d8618dd4d42e1ca
checksum/health: a5073935c8eb985cf8f3128ba7abbc4121cef628a9a1b0924c95cf97d33323bf
checksum/secret: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
cluster-autoscaler.kubernetes.io/safe-to-evict: true
kubernetes.io/psp: eks.privileged
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/airflow-redis-master
NominatedNodeName: 6f344dfd11-000a9c54e4e240a2a8b3dfceb5f8227e
Containers:
airflow-redis:
Image: docker.io/bitnami/redis:5.0.7-debian-10-r32
Port: 6379/TCP
Host Port: 0/TCP
Command:
/bin/bash
-c
if [[ -n $REDIS_PASSWORD_FILE ]]; then
password_aux=`cat ${REDIS_PASSWORD_FILE}`
export REDIS_PASSWORD=$password_aux
fi
if [[ ! -f /opt/bitnami/redis/etc/master.conf ]];then
cp /opt/bitnami/redis/mounted-etc/master.conf /opt/bitnami/redis/etc/master.conf
fi
if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then
cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf
fi
ARGS=("--port" "${REDIS_PORT}")
ARGS+=("--requirepass" "${REDIS_PASSWORD}")
ARGS+=("--masterauth" "${REDIS_PASSWORD}")
ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf")
ARGS+=("--include" "/opt/bitnami/redis/etc/master.conf")
/run.sh ${ARGS[#]}
Liveness: exec [sh -c /health/ping_liveness_local.sh 5] delay=5s timeout=5s period=5s #success=1 #failure=5
Readiness: exec [sh -c /health/ping_readiness_local.sh 5] delay=5s timeout=1s period=5s #success=1 #failure=5
Environment:
REDIS_REPLICATION_MODE: master
REDIS_PASSWORD: <set to the key 'redis-password' in secret 'my-creds'> Optional: false
REDIS_PORT: 6379
Mounts:
/data from redis-data (rw)
/health from health (rw)
/opt/bitnami/redis/etc/ from redis-tmp-conf (rw)
/opt/bitnami/redis/mounted-etc from config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-dmwvn (ro)
Volumes:
health:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-redis-health
Optional: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-redis
Optional: false
redis-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
redis-tmp-conf:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
default-token-dmwvn:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-dmwvn
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal LoggingEnabled 3m12s fargate-scheduler Successfully enabled logging for pod
Warning FailedScheduling 12s fargate-scheduler Pod provisioning timed out (will retry) for pod: airflow/airflow-redis-master-0
Other things I have tried
Tagging my subnets with the appropriate tag (conditional based on public/private subnets):
kubernetes_tags = map(
"kubernetes.io/role/${var.type == "Public" ? "elb" : "internal-elb"}", 1,
"kubernetes.io/cluster/${var.kubernetes_cluster_name}", "shared"
)
Annotating my pods with the Fargate profile (like infrastructure:fargate)
Debug VPC settings. To my understanding, the following settings need to be described for Fargate (source here):
single_nat_gateway = true # needed for fargate (https://docs.aws.amazon.com/eks/latest/userguide/eks-ug.pdf#page=135&zoom=100,96,764)
enable_nat_gateway = true # needed for fargate (https://docs.aws.amazon.com/eks/latest/userguide/eks-ug.pdf#page=135&zoom=100,96,764)
enable_vpn_gateway = false
enable_dns_hostnames = true # needed for fargate (https://docs.aws.amazon.com/eks/latest/userguide/eks-ug.pdf#page=135&zoom=100,96,764)
enable_dns_support = true # needed for fargate (https://docs.aws.amazon.com/eks/latest/userguide/eks-ug.pdf#page=135&zoom=100,96,764)
However, I have been provided a readily created VPC, and I am not sure how to check if these settings have been already turned on/off.
What are the steps that I need to take to debug this issue?
For test purposes I think you need to enable connectivity from the vpc private subnets to the outside world using NAT gateway.
So you may create NAT gateway in Public and add to the private subnets additional entry in their associated Routing table that looks like this:
0.0.0.0/0 nat-xxxxxxxx
if this worked and you want to keep your outbound restricted through your firewall instance which is more secure,
I think you need to contact firewall provider support to ask how you can whitelist farget outound traffic.
I am using the official helm chart of Jenkins.
I have enabled backup and also provided backup credentials
Here is the relevant config in values.yaml
## Backup cronjob configuration
## Ref: https://github.com/maorfr/kube-tasks
backup:
# Backup must use RBAC
# So by enabling backup you are enabling RBAC specific for backup
enabled: true
# Used for label app.kubernetes.io/component
componentName: "jenkins-backup"
# Schedule to run jobs. Must be in cron time format
# Ref: https://crontab.guru/
schedule: "0 2 * * *"
labels: {}
annotations: {}
# Example for authorization to AWS S3 using kube2iam
# Can also be done using environment variables
# iam.amazonaws.com/role: "jenkins"
image:
repository: "maorfr/kube-tasks"
tag: "0.2.0"
# Additional arguments for kube-tasks
# Ref: https://github.com/maorfr/kube-tasks#simple-backup
extraArgs: []
# Add existingSecret for AWS credentials
existingSecret: {}
# gcpcredentials: "credentials.json"
## Example for using an existing secret
# jenkinsaws:
## Use this key for AWS access key ID
awsaccesskey: "AAAAJJJJDDDDDDJJJJJ"
## Use this key for AWS secret access key
awssecretkey: "frkmfrkmrlkmfrkmflkmlm"
# Add additional environment variables
# jenkinsgcp:
## Use this key for GCP credentials
env: []
# Example environment variable required for AWS credentials chain
# - name: "AWS_REGION"
# value: "us-east-1"
resources:
requests:
memory: 1Gi
cpu: 1
limits:
memory: 1Gi
cpu: 1
# Destination to store the backup artifacts
# Supported cloud storage services: AWS S3, Minio S3, Azure Blob Storage, Google Cloud Storage
# Additional support can added. Visit this repository for details
# Ref: https://github.com/maorfr/skbn
destination: "s3://jenkins-data/backup"
However the backup job fails as follows:
2020/01/22 20:19:23 Backup started!
2020/01/22 20:19:23 Getting clients
2020/01/22 20:19:26 NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
What is missing?
you must create secret which looks like this:
kubectl create secret generic jenkinsaws --from-literal=jenkins_aws_access_key=ACCESS_KEY --from-literal=jenkins_aws_secret_key=SECRET_KEY
then consume it like this:
existingSecret:
jenkinsaws:
awsaccesskey: jenkins_aws_access_key
awssecretkey: jenkins_aws_secret_key
where jenkins_aws_access_key/jenkins_aws_secret_key it's key of the secret
backup:
enabled: true
destination: "s3://jenkins-pumbala/backup"
schedule: "15 1 * * *"
env:
- name: "AWS_ACCESS_KEY_ID"
value: "AKIDFFERWT***D36G"
- name: "AWS_SECRET_ACCESS_KEY"
value: "5zGdfgdfgdf***************Isi"