Our company is using AWS and we have deployed an OpenShift OKD cluster using openshift-installer and following the instructions on the page "Installing a private cluster on AWS". We have been using the cluster for a while and everything has been going well.
Recently, I need to expose some Services using the NodePort in addition to the usual HTTP ports (80, 443), specifically the range 30000-32767. I discover that the Installer deployed a (private) Classic Load Balancer with only two Listeners for port 80 and 443 which to me was sensible.
I have manually added several more listeners for the NodePort ports and they are working as expected.
cat <<'__ELB__' | xargs -0 aws elb create-load-balancer-listeners --cli-input-yaml
LoadBalancerName: 'abcdefghijklmnopqrstu012345678'
Listeners:
- Protocol: 'TCP'
LoadBalancerPort: 32348
InstanceProtocol: 'TCP'
InstancePort: 32348
#- ... more listeners omitted
__ELB__
However, after a few days, I noticed that those added listeners were removed. By checking CloudTrail history, it turned out that the listeners were removed by one of the Control Plane.
// Please note that all information were redacted as well as irrelevant properties
// were removed
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROAQO2JUSTEXAMPLE:i-03b1248d0example",
"arn": "arn:aws:sts::0123456789:assumed-role/ExampleOrg__OKD4--ControlPlane/i-03b1248d0example",
"accountId": "0123456789",
"accessKeyId": "ASIAABCDEEXAMPLEONLY",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROAQO2JUSTEXAMPLE",
"arn": "arn:aws:iam::0123456789:role/ExampleOrg__OKD4--ControlPlane",
"accountId": "0123456789",
"userName": "ExampleOrg__OKD4--ControlPlane"
},
"ec2RoleDelivery": "2.0"
}
},
"eventSource": "elasticloadbalancing.amazonaws.com",
"eventName": "DeleteLoadBalancerListeners",
"userAgent": "kubernetes/v1.23.3-2003+e419edff267ffa-dirty aws-sdk-go/1.38.49 (go1.17.5; linux; amd64)",
"requestParameters": {
"loadBalancerName": "abcdefghijklmnopqrstu012345678",
"loadBalancerPorts": [
32349,
...
]
},
"responseElements": null,
"eventType": "AwsApiCall",
"apiVersion": "2012-06-01",
"managementEvent": true,
"eventCategory": "Management"
}
There were other actions from the Control Plane which modified the Load Balancers.
I tried to search the logs of the Pods of the Operators on the Control Planes as well as the Git repositories on GitHub 2 but I could not find any hint where these calls to AWS were made.
I really appreciate if anyone who could point me to:
Which operators/components on OKD/OpenShift are used to update the AWS resources given the fact that the cluster was installed using Mint mode and cloud credential (aws-creds) has not been removed?
Is it possible that the Control Planes themselves (EC2 instance) with IAM Role make those calls outside the OpenShift cluster (like other daemon processes)?
What would be the correct way to amend the ports into the cluster?
Related
I am currently configuring my production instances to use AWS Backup Service rather than Lamdba. However, I notice AWS Backup Service does not have the "no reboot" option or anything mentioning that it will not reboot the EC2 instances.
Hence, Will AWS Backup Service restart my EC2 instances during the backup(create AMI) process?
It will not reboot your instance. I checked that using on-demand backup of my instance. Then in CloudTrial I verified that the CreateImage API call made by the backup is set with "noReboot": true:
From CloudTrial event (part shown):
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "xxxx:AWSBackup-AWSBackupDefaultServiceRole",
"arn": "arn:aws:sts::xxxx:assumed-role/AWSBackupDefaultServiceRole/AWSBackup-AWSBackupDefaultServiceRole",
"eventSource": "ec2.amazonaws.com",
"eventName": "CreateImage",
"requestParameters": {
"description": "This image is created by the AWS Backup service.",
"noReboot": true
},
}
What is the easiest way to provide one or several external configuration file(s) to an app running as an AWS Fargate task?
The files cannot be part of the Docker image because they depend on the stage environment and may contain secrets.
Creating an EFS volume just for this seems to be overengineered (we only need read access to some kb of properties).
Using the AWS SDK to access a S3 bucket at startup means that the app has a dependency to the SDK, and one has to manage S3 buckets.*
Using AWS AppConfig would still require the app to use the AWS SDK to access the config values.*
Having hundreds of key-value pairs in the Parameter Store would be ugly.
*It is an important aspect of our applications to not depend on the AWS SDK, because we need to be able to deploy to different cloud platforms, so solutions that avoid this are preferable.
It would be nice to just be able to define this in the task definition, so that Fargate mounts a couple of files in the container. Is this or a similar low-key solution available?
There's a specific feature of AWS Systems Manager for that purpose, called AWS AppConfig. It helps you deploy application configuration just like code deployments, but without the need to re-deploy the code if a configuration value changes.
The following article illustrates the integration between containers and AWS AppConfig: Application configuration deployment to container workloads using AWS AppConfig.
Not an answer to the question but in case someone comes here looking for solutions, we had the same requirements but did not find an easy solution to deploy configuration file directly in ECS instance for the container to read. I'm sure it's possible, just would make is difficult to configure, therefore we did not see the effort worthy.
What we did:
Added EnvironmentConfigBuilder as discribed in MS docs here
Passed in configuration values using environment variables as described in AWS docs here.
You can specify your AWS AppConfig dependency as a separate container. AWS gives you the option to set container dependency execution conditions in your Task Definition. See: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerDependency.html
You could set your container dependency status to COMPLETE for the container that pulls the config files from AppConfig and then just treat the files as a dumb mount, separating the AWS dependency completely. For Example:
"containerDefinitions": [
{
"name": "app-config-script",
"image": "1234567890.dkr.ecr.SOME_REGION.amazonaws.com/app-config-script:ver",
"essential": false,
"mountPoints": [
{
"sourceVolume": "config",
"containerPath": "/data/config/nginx",
"readOnly": ""
}
],
"dependsOn": null,
"repositoryCredentials": {
"credentialsParameter": ""
}
},
{
"name": "nginx",
"image": "nginx",
"essential": true,
"portMappings": [
{
"containerPort": "80",
"protocol": "tcp"
},
{
"containerPort": "443",
"protocol": "tcp"
}
],
"mountPoints": [
{
"sourceVolume": "config",
"containerPath": "/etc/nginx",
"readOnly": true
}
],
"dependsOn": [
{
"containerName": "app-config-script",
"condition": "COMPLETE"
}
],
"repositoryCredentials": {
"credentialsParameter": ""
}
}
],
Your Entrypoint/CMD script in the bootstrap container would then be something like:
#!/bin/sh
token=$(aws appconfigdata start-configuration-session --application-identifier "${APPLICATION_ID}" --environment-identifier "${ENVIRONMENT_ID}" --configuration-profile-identifier "${CONFIGURATION_ID}" | jq -r .InitialConfigurationToken)
aws appconfigdata get-latest-configuration --configuration-token "${token}" /data/config/nginx/nginx.conf
We are using Packer to build images in a GCP compute instance. Packer tries to grab the image based on project and image as follows:
https://www.googleapis.com/compute/v1/projects/<project-name>/global/images/<image-name>?alt=json
Then it throws an error:
oauth2: cannot fetch token: Post https://accounts.google.com/o/oauth2/token: dial tcp 108.177.111.84:443: i/o timeout
Based on security principle, our compute instance has no external IP address, therefore it does not have access to internet. In this case, accounts.google.com is no longer accessible. Then how can we authenticate google apis?
I tried to enable firewall rules and provide routes for internet access. But based on the requirement stated here, the instance still won't have access if it does not have external IP address.
This means we must have a separate way to authenticate googleapis.
But does Packer support this?
Here is the packer builder we have:
"builders": [
{
"type": "googlecompute",
"project_id": "test",
"machine_type": "n1-standard-4",
"source_image_family": "{{user `source_family`}}",
"source_image": "{{user `source_image`}}",
"source_image_project_id": "{{user `source_project_id`}}",
"region": "{{user `region`}}",
"zone": "{{user `zone`}}",
"network": "{{user `network`}}",
"subnetwork": "{{user `subnetwork`}}",
"image_name": "test-{{timestamp}}",
"disk_size": 10,
"disk_type": "pd-ssd",
"state_timeout": "5m",
"ssh_username": "build",
"ssh_timeout": "1000s",
"ssh_private_key_file": "./gcp-instance-key.pem",
"service_account_email": "test-account#test-mine.iam.gserviceaccount.com",
"omit_external_ip": true,
"use_internal_ip": true,
"metadata": {
"user": "build"
}
}
To do what you want manually you will need to have an ssh tunnel open on a working compute instance inside the project or in a vpc that has a peering enabled on the network the compute you want to reach is.
If you then use a CI with a runner like gitlab-ci, be sure to create the runner inside the same vpc or in a vpc with a peering.
If you don't want to create a compute with an external ip you could try to open a vpn connection to the project and do it through the vpn.
I have a Java webapp deployed in ECS using the tomcat:8.5-jre8-alpine image. The network mode for this task is awsvpc; I have many of these tasks running across 3 EC2 instances fronted by an ALB.
This is working fine but now I want to add an nginx reverse-proxy in front of each tomcat container, similar to this example: https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy.
My abbreviated container definition file is:
{
"containerDefinitions": [
{
"name": "nginx",
"image": "<NGINX reverse proxy image URL>",
"memory": "256",
"cpu": "256",
"essential": true,
"portMappings": [
{
"containerPort": "80",
"protocol": "tcp"
}
],
"links": [
"app"
]
},
{
"name": "app",
"image": "<app image URL>",
"memory": "1024",
"cpu": "1024",
"essential": true
}
],
"volumes": [],
"networkMode": "awsvpc",
"placementConstraints": [],
"family": "application-stack"
}
When I try to save a new task definition I received the error: "links are not supported when the network type is awsvpc"
I am using the awsvpc network mode because it gives me granular control over the inbound traffic via a security group.
Is there any way to create a task definition with 2 linked containers when using awsvpc network mode?
You dont need the linking part at all, because awsvpc allows you to reference other containers simply by using
localhost:8080
(or whatever port is your other container mapped to)
in your nginx config file.
So remove links from your json and use localhost:{container port} in nginx config. Simple as that.
Actually if you want to use a reverse-proxy you can stop using links, because you can make service discovery or using your reverse-proxy to use your dependency.
If you still want to use link instead of using that reverse proxy you can use consul and Fabio. Both services are dockerizable.
With this, there is no necessity to use awsvpc and you can use consul for service-discovery.
Hope it helps!
I am trying to access the host port and set it as dockerLabels while defining the Cloud Formation some thing like below:
"dockerLabels": {"SERVICE_PORT": "${service_port}"},
"portMappings": [
{
"containerPort": {
"Ref": "ContainerPort"
},
"hostPort": ${service_port}
}
],
I was trying to access these ports later through aws-sdk for service discovery.
Service ports are dynamic and Cloudformation doesn't keep up with ever-changing values. If you can work with static ports, then declare it as a cfn parameter and reference it in your snippet. You can then use that port in your service discovery setup