I'm currently working through some GCP training courses via https://www.cloudskillsboost.google/ and seeing some differences in my output versus the documentation. Not sure if this is down to the software version, account settings or what?
Example given is for a list of firewall rules:
gcloud compute firewall-rules list
and the example output is
NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED
default-allow-icmp default INGRESS 65534 icmp False
default-allow-internal default INGRESS 65534 tcp:0-65535,udp:0-65535,icmp False
default-allow-rdp default INGRESS 65534 tcp:3389 False
default-allow-ssh default INGRESS 65534 tcp:22 False
dev-net-allow-ssh dev-network INGRESS 1000 tcp:22 False
serverless-to-vpc-connector dev-network INGRESS 1000 icmp,udp:665-666,tcp:667 False
vpc-connector-egress dev-network INGRESS 1000 icmp,udp,tcp False
vpc-connector-health-check dev-network INGRESS 1000 tcp:667 False
vpc-connector-to-serverless dev-network EGRESS 1000 icmp,udp:665-666,tcp:667 False
One rule per line with multiple fields horizontally. However, when I run this command in the Cloud Shell I get the following:
NAME: default-allow-icmp
NETWORK: default
DIRECTION: INGRESS
PRIORITY: 65534
ALLOW: icmp
DENY:
DISABLED: False
NAME: default-allow-internal
NETWORK: default
DIRECTION: INGRESS
PRIORITY: 65534
ALLOW: tcp:0-65535,udp:0-65535,icmp
DENY:
DISABLED: False
NAME: default-allow-rdp
NETWORK: default
DIRECTION: INGRESS
PRIORITY: 65534
ALLOW: tcp:3389
DENY:
DISABLED: False
NAME: default-allow-ssh
NETWORK: default
DIRECTION: INGRESS
PRIORITY: 65534
ALLOW: tcp:22
DENY:
DISABLED: False
NAME: fw-allow-health-check
NETWORK: default
DIRECTION: INGRESS
PRIORITY: 1000
ALLOW: tcp:80
DENY:
DISABLED: False
NAME: www-firewall-network-lb
NETWORK: default
DIRECTION: INGRESS
PRIORITY: 1000
ALLOW: tcp:80
DENY:
DISABLED: False
It's the same data but the layout is different. (Appreciate that output is not identical, but I'm ot concerned abiout that. Differences between sample and live data is ok here I think)
Now, you can change the output format with the format option, but I can't recreate this tab separated table.
Is this just down to the documentation reformatting the output? Is it down to the software version or a setting I've missed? Can the format option do this and I've just not figured it out?
Any pointers much appreciated, as it's frustrating me.
Sharing you the best practices document [1], which explains the ways to format with example scripts.
You can check the possible outputs of --format in the [2] and also a similar thread [SO link] explains the ways to generate excel format output.
Related
AWS CDK lib version: 2.62.2
When I let the CDK implicitly create the SecurityGroup as below:
this.alb = new ApplicationLoadBalancer(this, 'Alb', {
loadBalancerName: "Raido",
vpc: props.vpc,
internetFacing: true,
vpcSubnets: {
subnetType: SubnetType.PUBLIC,
onePerAz: true,
},
// securityGroup: props.securityGroup,
idleTimeout: Duration.seconds(60),
dropInvalidHeaderFields: false,
desyncMitigationMode: DesyncMitigationMode.DEFENSIVE,
});
It creates an ICMP egress rule blocking port 86, like this:
egress rule:
Type: Custom ICMP - IPv4
Protocol: 252
Port: 86
Destination: 255.255.255.255/32
Description: Disallow all traffic
When I create my own SecurityGroup manually, it doesn't have the egress rule.
Why does the CDK create this egress rule, and should I add a similar rule of my own?
I'm reluctant to just copy/paste the rule without knowing what it's for.
Apparently, this rule is added when allowAllOutbound option is false, which is the default for the ApplicationLoadBalancer construct.
My takeaway: no point in setting the bogus rule on your ALB. If you want to allowAllOutbound, then do that; otherwise set your egress rules appropriate to your situation.
Apparenlty the 255.255.255.255/32 is intended to match no traffic because:
No machine can ever actually have the 255.255.255.255 IP address
I have a Java application running in a docker container. I am deploying all this to my GKE cluster. I'd like to have it connected to a CloudSQL instance via private IP. However I struggle for two days now to get it working. I followed the following guide:
https://cloud.google.com/sql/docs/mysql/configure-private-services-access#gcloud_1
I managed to create a private service connection and also gave my CloudSQL instance the address range. As far as I understood this should be sufficient for the Pod to be able to connect to the CloudSQL instance.
However it just does not work. I pass the private IP from CloudSQL as the host for the Java application JDBC (Database) connection.
│ 2022-02-14 22:03:31.299 WARN 1 --- [ main] o.h.e.j.e.i.JdbcEnvironmentInitiator : HHH000342: Could not obtain connection to query metadata │
│ │
│ com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
Here are some details to the problem.
The address allocation
➜ google-cloud-sdk gcloud compute addresses list
NAME ADDRESS/RANGE TYPE PURPOSE NETWORK REGION SUBNET STATUS
google-managed-services-default 10.77.0.0/16 INTERNAL VPC_PEERING default RESERVED
The vpc peering connection
➜ google-cloud-sdk gcloud services vpc-peerings list --network=default
---
network: projects/1071923183712/global/networks/default
peering: servicenetworking-googleapis-com
reservedPeeringRanges:
- google-managed-services-default
service: services/servicenetworking.googleapis.com
Here is my CloudSQL info. Please not that the PRIVATE IP Address is 10.77.0.5 and therefore matches the address range from above 10.77.0.0/16. I guess this part is working.
➜ google-cloud-sdk gcloud sql instances describe alpha-3
backendType: SECOND_GEN
connectionName: barbarus:europe-west4:alpha-3
createTime: '2022-02-14T19:28:02.465Z'
databaseInstalledVersion: MYSQL_5_7_36
databaseVersion: MYSQL_5_7
etag: 758de240b161b946689e5732d8e71d396c772c0e03904c46af3b61f59b1038a0
gceZone: europe-west4-a
instanceType: CLOUD_SQL_INSTANCE
ipAddresses:
- ipAddress: 34.90.174.243
type: PRIMARY
- ipAddress: 10.77.0.5
type: PRIVATE
kind: sql#instance
name: alpha-3
project: barbarus
region: europe-west4
selfLink: https://sqladmin.googleapis.com/sql/v1beta4/projects/barbarus/instances/alpha-3
serverCaCert:
cert: |-
-----BEGIN CERTIFICATE-----
//...
-----END CERTIFICATE-----
certSerialNumber: '0'
commonName: C=US,O=Google\, Inc,CN=Google Cloud SQL Server CA,dnQualifier=d495898b-f6c7-4e2f-9c59-c02ccf2c1395
createTime: '2022-02-14T19:29:35.325Z'
expirationTime: '2032-02-12T19:30:35.325Z'
instance: alpha-3
kind: sql#sslCert
sha1Fingerprint: 3ee799b139bf335ef39554b07a5027c9319087cb
serviceAccountEmailAddress: p1071923183712-d99fsz#gcp-sa-cloud-sql.iam.gserviceaccount.com
settings:
activationPolicy: ALWAYS
availabilityType: ZONAL
backupConfiguration:
backupRetentionSettings:
retainedBackups: 7
retentionUnit: COUNT
binaryLogEnabled: true
enabled: true
kind: sql#backupConfiguration
location: us
startTime: 12:00
transactionLogRetentionDays: 7
dataDiskSizeGb: '10'
dataDiskType: PD_HDD
ipConfiguration:
allocatedIpRange: google-managed-services-default
ipv4Enabled: true
privateNetwork: projects/barbarus/global/networks/default
requireSsl: false
kind: sql#settings
locationPreference:
kind: sql#locationPreference
zone: europe-west4-a
pricingPlan: PER_USE
replicationType: SYNCHRONOUS
settingsVersion: '1'
storageAutoResize: true
storageAutoResizeLimit: '0'
tier: db-f1-micro
state: RUNNABLE
The problem I see is with the Pod's IP Address. It is 10.0.5.3 and that is not in the range of 10.77.0.0/16 and therefore the pod can't see the CloudSQL instance.
See here is the Pod's info:
Name: game-server-5b9dd47cbd-vt2gw
Namespace: default
Priority: 0
Node: gke-barbarus-node-pool-1a5ea7d5-bg3m/10.164.15.216
Start Time: Tue, 15 Feb 2022 00:33:56 +0100
Labels: app=game-server
app.kubernetes.io/managed-by=gcp-cloud-build-deploy
pod-template-hash=5b9dd47cbd
Annotations: <none>
Status: Running
IP: 10.0.5.3
IPs:
IP: 10.0.5.3
Controlled By: ReplicaSet/game-server-5b9dd47cbd
Containers:
game-server:
Container ID: containerd://57d9540b1e5f5cb3fcc4517fa42377282943d292ba810c83cd7eb50bd4f1e3dd
Image: eu.gcr.io/barbarus/game-server#sha256:72d518a53652d32d0d438d2a5443c44cc8e12bb15cb1a59c843ce72466900141
Image ID: eu.gcr.io/barbarus/game-server#sha256:72d518a53652d32d0d438d2a5443c44cc8e12bb15cb1a59c843ce72466900141
Port: <none>
Host Port: <none>
State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 15 Feb 2022 00:36:48 +0100
Finished: Tue, 15 Feb 2022 00:38:01 +0100
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 15 Feb 2022 00:35:23 +0100
Finished: Tue, 15 Feb 2022 00:36:35 +0100
Ready: False
Restart Count: 2
Environment:
SQL_CONNECTION: <set to the key 'SQL_CONNECTION' of config map 'game-server'> Optional: false
SQL_USER: <set to the key 'SQL_USER' of config map 'game-server'> Optional: false
SQL_DATABASE: <set to the key 'SQL_DATABASE' of config map 'game-server'> Optional: false
SQL_PASSWORD: <set to the key 'SQL_PASSWORD' of config map 'game-server'> Optional: false
LOG_LEVEL: <set to the key 'LOG_LEVEL' of config map 'game-server'> Optional: false
WORLD_ID: <set to the key 'WORLD_ID' of config map 'game-server'> Optional: false
WORLD_SIZE: <set to the key 'WORLD_SIZE' of config map 'game-server'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sknlk (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-sknlk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m7s default-scheduler Successfully assigned default/game-server-5b9dd47cbd-vt2gw to gke-barbarus-node-pool-1a5ea7d5-bg3m
Normal Pulling 4m6s kubelet Pulling image "eu.gcr.io/barbarus/game-server#sha256:72d518a53652d32d0d438d2a5443c44cc8e12bb15cb1a59c843ce72466900141"
Normal Pulled 3m55s kubelet Successfully pulled image "eu.gcr.io/barbarus/game-server#sha256:72d518a53652d32d0d438d2a5443c44cc8e12bb15cb1a59c843ce72466900141" in 11.09487284s
Normal Created 75s (x3 over 3m54s) kubelet Created container game-server
Normal Started 75s (x3 over 3m54s) kubelet Started container game-server
Normal Pulled 75s (x2 over 2m41s) kubelet Container image "eu.gcr.io/barbarus/game-server#sha256:72d518a53652d32d0d438d2a5443c44cc8e12bb15cb1a59c843ce72466900141" already present on machine
Warning BackOff 1s (x2 over 87s) kubelet Back-off restarting failed container
Finally this is what gcloud container clusters describe gives me:
➜ google-cloud-sdk gcloud container clusters describe --region=europe-west4 barbarus
addonsConfig:
gcePersistentDiskCsiDriverConfig:
enabled: true
kubernetesDashboard:
disabled: true
networkPolicyConfig:
disabled: true
autopilot: {}
autoscaling:
autoscalingProfile: BALANCED
binaryAuthorization: {}
clusterIpv4Cidr: 10.0.0.0/14
createTime: '2022-02-14T19:34:03+00:00'
currentMasterVersion: 1.21.6-gke.1500
currentNodeCount: 3
currentNodeVersion: 1.21.6-gke.1500
databaseEncryption:
state: DECRYPTED
endpoint: 34.141.141.150
id: 39e7249b48c24d23a8b70b0c11cd18901565336b397147dab4778dc75dfc34e2
initialClusterVersion: 1.21.6-gke.1500
initialNodeCount: 1
instanceGroupUrls:
- https://www.googleapis.com/compute/v1/projects/barbarus/zones/europe-west4-a/instanceGroupManagers/gke-barbarus-node-pool-e291e3d6-grp
- https://www.googleapis.com/compute/v1/projects/barbarus/zones/europe-west4-b/instanceGroupManagers/gke-barbarus-node-pool-5aa35c39-grp
- https://www.googleapis.com/compute/v1/projects/barbarus/zones/europe-west4-c/instanceGroupManagers/gke-barbarus-node-pool-380645b7-grp
ipAllocationPolicy:
useRoutes: true
labelFingerprint: a9dc16a7
legacyAbac: {}
location: europe-west4
locations:
- europe-west4-a
- europe-west4-b
- europe-west4-c
loggingConfig:
componentConfig:
enableComponents:
- SYSTEM_COMPONENTS
- WORKLOADS
loggingService: logging.googleapis.com/kubernetes
maintenancePolicy:
resourceVersion: e3b0c442
masterAuth:
clusterCaCertificate: // ...
masterAuthorizedNetworksConfig: {}
monitoringConfig:
componentConfig:
enableComponents:
- SYSTEM_COMPONENTS
monitoringService: monitoring.googleapis.com/kubernetes
name: barbarus
network: default
networkConfig:
defaultSnatStatus: {}
network: projects/barbarus/global/networks/default
serviceExternalIpsConfig: {}
subnetwork: projects/barbarus/regions/europe-west4/subnetworks/default
nodeConfig:
diskSizeGb: 100
diskType: pd-standard
imageType: COS_CONTAINERD
machineType: e2-medium
metadata:
disable-legacy-endpoints: 'true'
oauthScopes:
- https://www.googleapis.com/auth/cloud-platform
preemptible: true
serviceAccount: default#barbarus.iam.gserviceaccount.com
shieldedInstanceConfig:
enableIntegrityMonitoring: true
nodeIpv4CidrSize: 24
nodePoolDefaults:
nodeConfigDefaults: {}
nodePools:
- config:
diskSizeGb: 100
diskType: pd-standard
imageType: COS_CONTAINERD
machineType: e2-medium
metadata:
disable-legacy-endpoints: 'true'
oauthScopes:
- https://www.googleapis.com/auth/cloud-platform
preemptible: true
serviceAccount: default#barbarus.iam.gserviceaccount.com
shieldedInstanceConfig:
enableIntegrityMonitoring: true
initialNodeCount: 1
instanceGroupUrls:
- https://www.googleapis.com/compute/v1/projects/barbarus/zones/europe-west4-a/instanceGroupManagers/gke-barbarus-node-pool-e291e3d6-grp
- https://www.googleapis.com/compute/v1/projects/barbarus/zones/europe-west4-b/instanceGroupManagers/gke-barbarus-node-pool-5aa35c39-grp
- https://www.googleapis.com/compute/v1/projects/barbarus/zones/europe-west4-c/instanceGroupManagers/gke-barbarus-node-pool-380645b7-grp
locations:
- europe-west4-a
- europe-west4-b
- europe-west4-c
management:
autoRepair: true
autoUpgrade: true
name: node-pool
podIpv4CidrSize: 24
selfLink: https://container.googleapis.com/v1/projects/barbarus/locations/europe-west4/clusters/barbarus/nodePools/node-pool
status: RUNNING
upgradeSettings:
maxSurge: 1
version: 1.21.6-gke.1500
notificationConfig:
pubsub: {}
releaseChannel:
channel: REGULAR
selfLink: https://container.googleapis.com/v1/projects/barbarus/locations/europe-west4/clusters/barbarus
servicesIpv4Cidr: 10.3.240.0/20
shieldedNodes:
enabled: true
status: RUNNING
subnetwork: default
zone: europe-west4
I have no idea how I can give the pod a reference to the address allocation I made for the private service connection.
What I tried is to spin up a GKE cluster with a Cluster default pod address range of 10.77.0.0/16 which sounded logical since I want the pods to appear in the same address range as the CloudSQL. However GCP gives me an error when I try to do that:
(1) insufficient regional quota to satisfy request: resource "CPUS": request requires '9.0' and is short '1.0'. project has a quota of '8.0' with '8.0' available. View and manage quotas at https://console.cloud.google.com/iam-admin/quotas?usage=USED&project=hait-barbarus (2) insufficient regional quota to satisfy request: resource "IN_USE_ADDRESSES": request requires '9.0' and is short '5.0'. project has a quota of '4.0' with '4.0' available. View and manage quotas at https://console.cloud.google.com/iam-admin/quotas?usage=USED&project=hait-barbarus.
So I am not able to give the pods the proper address range for the private service connection how can they ever discover the CloudSQL instance?
EDIT #1: The GKE cluster's service account has the SQL Client role.
Objective
My objective is to be able to deploy on AWS EKS using Fargate. I have successfully made the deployment work with a node_group. However, when I shifted to using Fargate, it seems that the pods are all stuck in the pending state.
How my current code looks like
I am provisioning using Terraform (not necessarily looking for a Terraform answer). This is how I create my EKS Cluster:
module "eks_cluster" {
source = "terraform-aws-modules/eks/aws"
version = "13.2.1"
cluster_name = "${var.project_name}-${var.env_name}"
cluster_version = var.cluster_version
vpc_id = var.vpc_id
cluster_enabled_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
enable_irsa = true
subnets = concat(var.private_subnet_ids, var.public_subnet_ids)
create_fargate_pod_execution_role = false
node_groups = {
my_nodes = {
desired_capacity = 1
max_capacity = 2
min_capacity = 1
instance_type = var.nodes_instance_type
subnets = var.private_subnet_ids
}
}
}
And this is how I provision the Fargate profile:
resource "aws_eks_fargate_profile" "airflow" {
cluster_name = module.eks_cluster.cluster_id
fargate_profile_name = "${var.project_name}-fargate-${var.env_name}"
pod_execution_role_arn = aws_iam_role.fargate_iam_role.arn
subnet_ids = var.private_subnet_ids
selector {
namespace = "airflow"
}
}
And this is how I created and attach the required policies:
resource "aws_iam_role" "fargate_iam_role" {
name = "${var.project_name}-fargate-${var.env_name}"
force_detach_policies = true
assume_role_policy = jsonencode({
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = ["eks-fargate-pods.amazonaws.com", "eks.amazonaws.com"]
}
}]
Version = "2012-10-17"
})
}
# Attach IAM Policy for Fargate
resource "aws_iam_role_policy_attachment" "fargate_pod_execution" {
role = aws_iam_role.fargate_iam_role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSFargatePodExecutionRolePolicy"
}
What I have tried and does not work
I have tried deploying the pods (I am using a Helm chart) in the same namespace where the Fargate Profile Exists. When I run kubectl get pods -n airflow I see all my pods pending like:
NAME READY STATUS RESTARTS AGE
airflow-flower-79b5948677-vww5d 0/1 Pending 0 40s
airflow-redis-master-0 0/1 Pending 0 40s
airflow-scheduler-6b6bd4b6f6-j9qzg 0/2 Pending 0 41s
airflow-web-567b55fbbf-z8dsg 0/2 Pending 0 41s
airflow-worker-0 0/2 Pending 0 40s
airflow-worker-1 0/2 Pending 0 40s
Then I look at the events by kubectl get events -n airflow, to which I get:
LAST SEEN TYPE REASON OBJECT MESSAGE
2m15s Normal LoggingEnabled pod/airflow-flower-79b5948677-vww5d Successfully enabled logging for pod
2m16s Normal SuccessfulCreate replicaset/airflow-flower-79b5948677 Created pod: airflow-flower-79b5948677-vww5d
2m17s Normal ScalingReplicaSet deployment/airflow-flower Scaled up replica set airflow-flower-79b5948677 to 1
2m15s Normal LoggingEnabled pod/airflow-redis-master-0 Successfully enabled logging for pod
2m16s Normal SuccessfulCreate statefulset/airflow-redis-master create Pod airflow-redis-master-0 in StatefulSet airflow-redis-master successful
2m15s Normal LoggingEnabled pod/airflow-scheduler-6b6bd4b6f6-j9qzg Successfully enabled logging for pod
2m16s Normal SuccessfulCreate replicaset/airflow-scheduler-6b6bd4b6f6 Created pod: airflow-scheduler-6b6bd4b6f6-j9qzg
2m17s Normal NoPods poddisruptionbudget/airflow-scheduler No matching pods found
2m17s Normal ScalingReplicaSet deployment/airflow-scheduler Scaled up replica set airflow-scheduler-6b6bd4b6f6 to 1
2m15s Normal LoggingEnabled pod/airflow-web-567b55fbbf-z8dsg Successfully enabled logging for pod
2m16s Normal SuccessfulCreate replicaset/airflow-web-567b55fbbf Created pod: airflow-web-567b55fbbf-z8dsg
2m17s Normal ScalingReplicaSet deployment/airflow-web Scaled up replica set airflow-web-567b55fbbf to 1
2m15s Normal LoggingEnabled pod/airflow-worker-0 Successfully enabled logging for pod
2m15s Normal LoggingEnabled pod/airflow-worker-1 Successfully enabled logging for pod
2m16s Normal SuccessfulCreate statefulset/airflow-worker create Pod airflow-worker-0 in StatefulSet airflow-worker successful
2m16s Normal SuccessfulCreate statefulset/airflow-worker create Pod airflow-worker-1 in StatefulSet airflow-worker successful
I then try to describe one of the pods (via kubectl describe pod), and I get:
Name: airflow-redis-master-0
Namespace: airflow
Priority: 2000001000
Priority Class Name: system-node-critical
Node: <none>
Labels: app=redis
chart=redis-10.5.7
controller-revision-hash=airflow-redis-master-588d57785d
eks.amazonaws.com/fargate-profile=airflow-fargate-airflow-dev
release=airflow
role=master
statefulset.kubernetes.io/pod-name=airflow-redis-master-0
Annotations: CapacityProvisioned: 0.25vCPU 0.5GB
Logging: LoggingEnabled
checksum/configmap: 2b82c78fd9186045e6e2b44cfbb38460310697cf2f2f175c9d8618dd4d42e1ca
checksum/health: a5073935c8eb985cf8f3128ba7abbc4121cef628a9a1b0924c95cf97d33323bf
checksum/secret: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
cluster-autoscaler.kubernetes.io/safe-to-evict: true
kubernetes.io/psp: eks.privileged
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/airflow-redis-master
NominatedNodeName: 6f344dfd11-000a9c54e4e240a2a8b3dfceb5f8227e
Containers:
airflow-redis:
Image: docker.io/bitnami/redis:5.0.7-debian-10-r32
Port: 6379/TCP
Host Port: 0/TCP
Command:
/bin/bash
-c
if [[ -n $REDIS_PASSWORD_FILE ]]; then
password_aux=`cat ${REDIS_PASSWORD_FILE}`
export REDIS_PASSWORD=$password_aux
fi
if [[ ! -f /opt/bitnami/redis/etc/master.conf ]];then
cp /opt/bitnami/redis/mounted-etc/master.conf /opt/bitnami/redis/etc/master.conf
fi
if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then
cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf
fi
ARGS=("--port" "${REDIS_PORT}")
ARGS+=("--requirepass" "${REDIS_PASSWORD}")
ARGS+=("--masterauth" "${REDIS_PASSWORD}")
ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf")
ARGS+=("--include" "/opt/bitnami/redis/etc/master.conf")
/run.sh ${ARGS[#]}
Liveness: exec [sh -c /health/ping_liveness_local.sh 5] delay=5s timeout=5s period=5s #success=1 #failure=5
Readiness: exec [sh -c /health/ping_readiness_local.sh 5] delay=5s timeout=1s period=5s #success=1 #failure=5
Environment:
REDIS_REPLICATION_MODE: master
REDIS_PASSWORD: <set to the key 'redis-password' in secret 'my-creds'> Optional: false
REDIS_PORT: 6379
Mounts:
/data from redis-data (rw)
/health from health (rw)
/opt/bitnami/redis/etc/ from redis-tmp-conf (rw)
/opt/bitnami/redis/mounted-etc from config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-dmwvn (ro)
Volumes:
health:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-redis-health
Optional: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-redis
Optional: false
redis-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
redis-tmp-conf:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
default-token-dmwvn:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-dmwvn
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal LoggingEnabled 3m12s fargate-scheduler Successfully enabled logging for pod
Warning FailedScheduling 12s fargate-scheduler Pod provisioning timed out (will retry) for pod: airflow/airflow-redis-master-0
Other things I have tried
Tagging my subnets with the appropriate tag (conditional based on public/private subnets):
kubernetes_tags = map(
"kubernetes.io/role/${var.type == "Public" ? "elb" : "internal-elb"}", 1,
"kubernetes.io/cluster/${var.kubernetes_cluster_name}", "shared"
)
Annotating my pods with the Fargate profile (like infrastructure:fargate)
Debug VPC settings. To my understanding, the following settings need to be described for Fargate (source here):
single_nat_gateway = true # needed for fargate (https://docs.aws.amazon.com/eks/latest/userguide/eks-ug.pdf#page=135&zoom=100,96,764)
enable_nat_gateway = true # needed for fargate (https://docs.aws.amazon.com/eks/latest/userguide/eks-ug.pdf#page=135&zoom=100,96,764)
enable_vpn_gateway = false
enable_dns_hostnames = true # needed for fargate (https://docs.aws.amazon.com/eks/latest/userguide/eks-ug.pdf#page=135&zoom=100,96,764)
enable_dns_support = true # needed for fargate (https://docs.aws.amazon.com/eks/latest/userguide/eks-ug.pdf#page=135&zoom=100,96,764)
However, I have been provided a readily created VPC, and I am not sure how to check if these settings have been already turned on/off.
What are the steps that I need to take to debug this issue?
For test purposes I think you need to enable connectivity from the vpc private subnets to the outside world using NAT gateway.
So you may create NAT gateway in Public and add to the private subnets additional entry in their associated Routing table that looks like this:
0.0.0.0/0 nat-xxxxxxxx
if this worked and you want to keep your outbound restricted through your firewall instance which is more secure,
I think you need to contact firewall provider support to ask how you can whitelist farget outound traffic.
Problem
We run Istio on our Kubernetes cluster and we're implementing AuthorizationPolicies.
We want to apply a filter on email address, an HTTP-condition only applicable to HTTP services.
Our Kiali service should be an HTTP service (it has an HTTP port, an HTTP listener, and even has HTTP conditions applied to its filters), and yet the AuthorizationPolicy does not work.
What gives?
Our setup
We have a management namespace with an ingressgateway (port 443), and a gateway+virtual service for Kiali.
These latter two point to the Kiali service in the kiali namespace.
Both the management and kiali namespace have a deny-all policy and an allow policy to make an exception for particular users.
(See AuthorizationPolicy YAMLs below.)
Authorization on the management ingress gateway works.
The ingress gateway has 3 listeners, all HTTP, and HTTP conditions are created and applied as you would expect.
You can visit its backend services other than Kiali if you're on the email list, and you cannot do so if you're not on the email list.
Authorization on the Kiali service does not work.
It has 99 listeners (!), including an HTTP listener on its configured 20001 port and its IP, but it does not work.
You cannot visit the Kiali service (due to the default deny-all policy).
The Kiali service has port 20001 enabled and named 'http-kiali', so the VirtualService should be ok with that. (See YAMls for service and virtual service below).
EDIT: it was suggested that the syntax of the email values matters.
I think that has been taken care of:
in the management namespace, the YAML below works as expected
in the kiali namespace, the same YAML fails to work as expected.
the empty brackets in the 'property(map[request.auth.claims[email]:{[brackets#test.com] []}])' message I think are the Values (present) and NotValues (absent), respectively, as per 'constructed internal model: &{Permissions:[{Properties:[map[request.auth.claims[email]:{Values:[brackets#test.com] NotValues:[]}]]}]}'
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: testpolicy-brackets
namespace: kiali
spec:
action: ALLOW
rules:
- when:
- key: source.namespace
values: ["brackets"]
- key: request.auth.claims[email]
values: ["brackets#test.com"]
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: testpolicy-yamllist
namespace: kiali
spec:
action: ALLOW
rules:
- when:
- key: source.namespace
values:
- list
- key: request.auth.claims[email]
values:
- list#test.com
debug rbac found authorization allow policies for workload [app=kiali,pod-template-hash=5c97c4bb66,security.istio.io/tlsMode=istio,service.istio.io/canonical-name=kiali,service.istio.io/canonical-revision=v1.16.0,version=v1.16.0] in kiali
debug rbac constructed internal model: &{Permissions:[{Services:[] Hosts:[] NotHosts:[] Paths:[] NotPaths:[] Methods:[] NotMethods:[] Ports:[] NotPorts:[] Constraints:[] AllowAll:true v1beta1:true}] Principals:[{Users:[] Names:[] NotNames:[] Group: Groups:[] NotGroups:[] Namespaces:[] NotNamespaces:[] IPs:[] NotIPs:[] RequestPrincipals:[] NotRequestPrincipals:[] Properties:[map[source.namespace:{Values:[brackets] NotValues:[]}] map[request.auth.claims[email]:{Values:[brackets#test.com] NotValues:[]}]] AllowAll:false v1beta1:true}]}
debug rbac generated policy ns[kiali]-policy[testpolicy-brackets]-rule[0]: permissions:<and_rules:<rules:<any:true > > > principals:<and_ids:<ids:<or_ids:<ids:<metadata:<filter:"istio_authn" path:<key:"source.principal" > value:<string_match:<safe_regex:<google_re2:<> regex:".*/ns/brackets/.*" > > > > > > > ids:<or_ids:<ids:<metadata:<filter:"istio_authn" path:<key:"request.auth.claims" > path:<key:"email" > value:<list_match:<one_of:<string_match:<exact:"brackets#test.com" > > > > > > > > > >
debug rbac ignored HTTP principal for TCP service: property(map[request.auth.claims[email]:{[brackets#test.com] []}])
debug rbac role skipped for no principals found
debug rbac found authorization allow policies for workload [app=kiali,pod-template-hash=5c97c4bb66,security.istio.io/tlsMode=istio,service.istio.io/canonical-name=kiali,service.istio.io/canonical-revision=v1.16.0,version=v1.16.0] in kiali
debug rbac constructed internal model: &{Permissions:[{Services:[] Hosts:[] NotHosts:[] Paths:[] NotPaths:[] Methods:[] NotMethods:[] Ports:[] NotPorts:[] Constraints:[] AllowAll:true v1beta1:true}] Principals:[{Users:[] Names:[] NotNames:[] Group: Groups:[] NotGroups:[] Namespaces:[] NotNamespaces:[] IPs:[] NotIPs:[] RequestPrincipals:[] NotRequestPrincipals:[] Properties:[map[source.namespace:{Values:[list] NotValues:[]}] map[request.auth.claims[email]:{Values:[list#test.com] NotValues:[]}]] AllowAll:false v1beta1:true}]}
debug rbac generated policy ns[kiali]-policy[testpolicy-yamllist]-rule[0]: permissions:<and_rules:<rules:<any:true > > > principals:<and_ids:<ids:<or_ids:<ids:<metadata:<filter:"istio_authn" path:<key:"source.principal" > value:<string_match:<safe_regex:<google_re2:<> regex:".*/ns/list/.*" > > > > > > > ids:<or_ids:<ids:<metadata:<filter:"istio_authn" path:<key:"request.auth.claims" > path:<key:"email" > value:<list_match:<one_of:<string_match:<exact:"list#test.com" > > > > > > > > > >
debug rbac ignored HTTP principal for TCP service: property(map[request.auth.claims[email]:{[list#test.com] []}])
debug rbac role skipped for no principals found
(Follows: a list of YAMLs mentioned above)
# Cluster AuthorizationPolicies
## Management namespace
Name: default-deny-all-policy
Namespace: management
API Version: security.istio.io/v1beta1
Kind: AuthorizationPolicy
Spec:
---
Name: allow-specified-email-addresses
Namespace: management
API Version: security.istio.io/v1beta1
Kind: AuthorizationPolicy
Spec:
Action: ALLOW
Rules:
When:
Key: request.auth.claims[email]
Values:
my.email#my.provider.com
---
## Kiali namespace
Name: default-deny-all-policy
Namespace: kiali
API Version: security.istio.io/v1beta1
Kind: AuthorizationPolicy
Spec:
---
Name: allow-specified-email-addresses
Namespace: kiali
API Version: security.istio.io/v1beta1
Kind: AuthorizationPolicy
Spec:
Action: ALLOW
Rules:
When:
Key: request.auth.claims[email]
Values:
my.email#my.provider.com
---
# Kiali service YAML
apiVersion: v1
kind: Service
metadata:
labels:
app: kiali
version: v1.16.0
name: kiali
namespace: kiali
spec:
clusterIP: 10.233.18.102
ports:
- name: http-kiali
port: 20001
protocol: TCP
targetPort: 20001
selector:
app: kiali
version: v1.16.0
sessionAffinity: None
type: ClusterIP
---
# Kiali VirtualService YAML
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: kiali-virtualservice
namespace: management
spec:
gateways:
- kiali-gateway
hosts:
- our_external_kiali_url
http:
- match:
- uri:
prefix: /
route:
- destination:
host: kiali.kiali.svc.cluster.local
port:
number: 20001
Marking as solved: I had forgotten to apply a RequestAuthentication to the Kiali namespace.
The problematic situation, with the fix in bold:
RequestAuthentication on the management namespace adds a user JWT (through an EnvoyFilter that forwards requests to an authentication service)
AuthorizationPolicy on the management namespace checks the request.auth.claims[email]. These fields exist in the JWT and all is well.
RequestAuthentication on the Kiali namespace missing. I fixed the problem by adding a RequestAuthentication for the Kiali namespace, which populates the user information, which allows the AuthorizationPolicy to perform its checks on actually existing fields.
AuthorizationPolicy on the Kiali namespace also checks the request.auth.claims[email] field, but since there is no authentication, there is no JWT with these fields. (There are some fields populated, e.g. source.namespace, but nothing like a JWT.) Hence, user validation on that field fails, as you would expect.
According to istio documentation:
Unsupported keys and values are silently ignored.
In your debug log there is:
debug rbac ignored HTTP principal for TCP service: property(map[request.auth.claims[email]:{[my.email#my.provider.com] []}])
As You can see there are []}] chars there that might suggest that the value got parsed the wrong way and got ignored as unsupported value.
Try to put Your values like suggested in documentation inside [""]:
request.auth.claims
Claims from the origin JWT. The actual claim name is surrounded by brackets HTTP only
key: request.auth.claims[iss]
values: ["*#foo.com"]
Hope it helps.
So I have an EKS cluster, and have set up the AWS Alb Ingress Controller:
https://github.com/kubernetes-sigs/aws-alb-ingress-controller
I'm trying to set up Grafana here, and the Ingress is created but it doesn't seem to resolve at all.
I have the follow Ingress:
$ kubectl describe ingress grafana
Name: grafana
Namespace: orbix-mvp
Address: 4ae1e4ba-orbixmvp-grafana-fd7d-993303634.eu-central-1.elb.amazonaws.com
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
grafana-orbix.orbixpay.com
/ grafana:80 (<none>)
Annotations:
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-2016-08
alb.ingress.kubernetes.io/subnets: subnet-08431d96168e36c30,subnet-0e2a7e2766852bf8a
alb.ingress.kubernetes.io/success-codes: 302
kubernetes.io/ingress.class: alb
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 45m alb-ingress-controller LoadBalancer 4ae1e4ba-orbixmvp-grafana-fd7d created, ARN: arn:aws:elasticloadbalancing:eu-central-1:109153834985:loadbalancer/app/4ae1e4ba-orbixmvp-grafana-fd7d/4b98cb7027b71697
Normal CREATE 45m alb-ingress-controller rule 1 created with conditions [{ Field: "host-header", Values: ["grafana-orbix.orbixpay.com"] },{ Field: "path-pattern", Values: ["/"] }]
The backend fro it is the following service:
$ kubectl describe service grafana
Name: grafana
Namespace: orbix-mvp
Labels: app=grafana
chart=grafana-1.25.1
heritage=Tiller
release=grafana
Annotations: <none>
Selector: app=grafana,release=grafana
Type: NodePort
IP: 172.20.11.232
Port: service 80/TCP
TargetPort: 3000/TCP
NodePort: service 30772/TCP
Endpoints: 10.0.0.180:3000
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
It does have a proper endpoint:
$ kubectl get endpoints | grep grafana
grafana 10.0.0.180:3000 46m
The pod itself is properly tagged and has the correct IP that's the endpoint above:
$ kubectl describe pod grafana-bdc977fd4-ptzhg
Name: grafana-bdc977fd4-ptzhg
Namespace: orbix-mvp
Priority: 0
PriorityClassName: <none>
Node: ip-10-0-0-230.eu-central-1.compute.internal/10.0.0.230
Start Time: Mon, 11 Feb 2019 13:24:43 +0200
Labels: app=grafana
pod-template-hash=687533980
release=grafana
Annotations: <none>
Status: Running
IP: 10.0.0.180
My AWS account has the LoadBalancer listed as Active, the subnets are on the same VPC as the cluster, security groups are being generated by the Ingress Controller.
Everything seems to be set up properly, however when I access the LoadBalancer address, it just times out.
$ kubectl get ingresses
NAME HOSTS ADDRESS PORTS AGE
grafana grafana-orbix.orbixpay.com 4ae1e4ba-orbixmvp-grafana-fd7d-993303634.eu-central-1.elb.amazonaws.com 80 49m
I actually figured it out - the Ingress configuration was allowing for traffic for the domain only. That excludes traffic to the load balancer address (which I assumed is allowed by default).
Basically it needs to be allowed for * in order for the Load Balancer URL to work too. Also, if the app redirects to /login like in my case, all paths need to be allowed too, since that redirect doesn't work if the path specified is for / only.