I have created an AWS VPC, subnet and security group and want to deploy my docker containers to these premade resources as a fargate ecs service.
However I don't know how to tell the service to use a premade subnet (it looks like it randomly picks a subnet from an 'allowed' list of subnets which is currently all subnets I have in my VPC
The below file correctly deploys to the desired vpc, cluster and security group, just not subnet:
version: '2'
x-aws-cluster: "Test Cluster"
x-aws-vpc: "vpc-02dffc2a8782579d4"
x-aws-security-group: "sg-02511658ffc184884"
services:
nginx:
image: nginx:1.19
networks:
- Backend-Access
networks:
Backend-Access:
external:
name: sg-02511658ffc184884
ipam:
driver: default
config:
- subnet: subnet-0aeef680f1f9e5cda # this has no effect
#- subnet: 172.31.4.0/24 also does not place the service in this subnet
I am running it using docker compose up -d (running it without -d gives a cluster does not exist error)
This is currently not possible and there is a GitHub issue open to ask for this specifically. If you could made your voice be heard there that's where we are consolidating this feedback. Thank you.
It is possible to make a subnet using the documented overlay method now and I solved it.
See the Github issue again! It was solved and closed two weeks ago.
Related
I am trying to deploy a web application in AWS fargate as well as AWS Beanstalk.
my docker compose file looks like this.(just an example , please focus on ports)
services:
application-gateway:
image: "gcr.io/docker-public/application:latest"
container_name: application-name
ports:
- "443:9443"
- "8443:8443"
**Issue with AWS Fargate
**
I need to know how to map these ports - Bridge doesnt get enabled and I see only
How to change Host Port
I can see that once I deploy the public docker image it gets deployed in Fargate however how to access the application DNS URL ?
**Issue facing in AWS Beanstalk
**
I was able to deploy the application in single instance however I am unable to deploy it in application load balanced enviroment. again I suspect the issue is with the ports in load balancer , I have opened these ports in security group though.
Thanks,
I am deploying a compose to an AWS ECS context with the following docker-compose.yml
x-aws-loadbalancer: "${LOADBALANCER_ARN}"
services:
webapi:
image: ${DOCKER_REGISTRY-}webapi
build:
context: .
dockerfile: webapi/Dockerfile
environment:
ASPNETCORE_URLS: http://+:80
ASPNETCORE_ENVIRONMENT: Development
ports:
- target: 80
x-aws-protocol: http
When I create a loadbalancer using these instructions the loadbalancer assigns the default security group for the default vpc. Which apparently doesn't match the ingress rules for the docker services because if I go and look at the task in ECS I see it being killed over and over for failing an ELB healtcheck.
The only way to fix it is to go into AWS Console and assign the created security group created by docker compose to represent the default network to the loadbalancer. But thats insane.
How do I create a loadbalancer with the correct minimum access security group so it will be able to talk to later created compose generated services?
I am deploying the following relatively simple docker-compose.yml file on AWS ECS via the Docker CLI.
It uses tomcat server image which can be also replaced by any other container which does not exits of startup.
services:
tomcat:
image: tomcat:9.0
command: catalina.sh run
ports:
- target: 8080
published: 8080
x-aws-protocol: http
Commands used
docker context use mycontextforecs
docker compose up
The cluster, services, task, target, security groups and application load balancer are automatically created as expected.
But, the security group created by AWS ECS allows inbound traffic on ALL ports by default instead of only the exposed 8080.
Following is a screenshot of the security group, which also has a comment -
"tomcat:8080/ on default network"
But port range is "All" instead of 8080
I've read the following and some other stackoverflow links but could not get an answer.
https://docs.docker.com/cloud/ecs-compose-features/
https://docs.docker.com/cloud/ecs-architecture/
https://docs.docker.com/cloud/ecs-integration/
I understand that the default "Fargate" instance type gets a public ip assigned.
But why does ECS allow traffic on all ports?
If I add another service in the docker-compose file, the default security group gets shared between both of them.
As a result, anyone can telnet into the port exposed by the service due to this security group rule.
Some info:
Kubernetes (1.5.1)
AWS
1 master and 1 node (both ubuntu 16.04)
k8s installed via kubeadm
Terraform made by me
Please don't reply use kube-up, kops or similar. This is about understanding how k8s works under the hood. There is by far too much unexplained magic in the system and I want to understand it.
== Question:
When creating a Service of type load balancer on k8s[aws] (for example):
apiVersion: v1
kind: Service
metadata:
name: kubernetes-dashboard
namespace: kube-system
labels:
k8s-addon: kubernetes-dashboard.addons.k8s.io
k8s-app: kubernetes-dashboard
kubernetes.io/cluster-service: "true"
facing: external
spec:
type: LoadBalancer
selector:
k8s-app: kubernetes-dashboard
ports:
- port: 80
I successfully create an internal or external facing ELB but none of the machines are added to the ELB (I can taint the master too but nothing changes). My problem is basically this:
https://github.com/kubernetes/kubernetes/issues/29298#issuecomment-260659722
The subnets and nodes (but not the VPC) are all tagged with "KubernetesCluster" (again... elb are created in the right place). However no nodes is added.
In the logs
kubectl logs kube-controller-manager-ip-x-x-x-x -n kube-system
after:
aws_loadbalancer.go:63] Creating load balancer for
kube-system/kubernetes-dashboard with name:
acd8acca0c7a111e69ca306f22de69ae
There is no other output (it should print the nodes added or removed). I tried to understand the code at:
https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws_loadbalancer.go
But whatever is the reason, this function to not add nodes.
The documentation doesn't go at length trying to explain the "process" behind k8s decisions. To try to understand k8s I tried/used kops, kube up, kubeadm, kubernetes the hard way repo and reading damn code, but still I am unable to understand how k8s on aws SELECTS the node to add to the elb.
As a consequence, also no security group is changed anywhere.
Is it a tag on the ec2?
Kublet setting?
Anything else?
Any help is greatly appreciated.
Thanks,
F.
I think Steve is on the right track. Make sure your kubelets, apiserver, and controller-manager components all include --cloud-provider=aws in their arguments lists.
You mention your subnets and instances all have matching KubernetesCluster tags. Do your controller & worker security groups? K8s will modify the worker SG in particular to allow traffic to/from the service ELBs it creates. I tag my VPC as well, though I guess it's not required and may prohibit another cluster from living in the same VPC.
I also tag my private subnets with kubernetes.io/role/internal-elb=true and public ones with kubernetes.io/role/elb=true to identify where internal and public ELBs can be created.
The full list (AFAIK) of tags and annotations lives in https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go
I think the node registration is being managed outside of Kubernetes. I'm using kops and if I edit the size of my ASG in AWS the new nodes are not registered with my service ELBs. But if I edit the number of nodes using kops the new nodes are there.
In the docs a kops instance group maps to an ASG when running on AWS. In the code it looks like its calling AWS rather than a k8s API.
I know you're not using kops but I think in Terraform you need to replicate the AWS API calls that kops is making.
Make sure you are setting the correct cloud provider settings with kubeadm (http://kubernetes.io/docs/admin/kubeadm/).
The AWS cloud provider automatically syncs the nodes available with the ELB. I created an type LoadBalancer then scaled my cluster and the new node was eventually added the ELB: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws_loadbalancer.go#L376
The CoreOS Multinode Cluster guide appears to have a problem. When I create a cluster and configure connectivity, everything appears fine -- however, I'm unable to create an ELB through service exposing:
$ kubectl expose rc my-nginx --port 80 --type=LoadBalancer
service "my-nginx" exposed
$ kubectl describe services
Name: my-nginx
Namespace: temp
Labels: run=my-nginx
Selector: run=my-nginx
Type: LoadBalancer
IP: 10.100.6.247
Port: <unnamed> 80/TCP
NodePort: <unnamed> 32224/TCP
Endpoints: 10.244.37.2:80,10.244.73.2:80
Session Affinity: None
No events.
The IP line that says 10.100.6.247 looks promising, but no ELB is actually created in my account. I can otherwise interact with the cluster just fine, so it seems bizarre. A "kubectl get services" listing is similar -- it shows the private IP (same as above) but the EXTERNAL_IP column is empty.
Ultimately, my goal is a solution that allows me to easily configure my VPC (ie. private subnets with NAT instances) and if I can get this working, it'd be easy enough to drop into CloudFormation since it's based on user-data. The official method of kube-up doesn't leave room for VPC-level customization in a repeatable way.
Unfortunately, that getting-started guide isn't nearly as up to date as the kube-up implementation. For instance, I don't see a --cloud-provider=aws flag anywhere, and the kubernetes-controller-manager would need that in order to know to call the AWS APIs.
You may want to check out the official CoreOS on AWS guide:
https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html
If you hit a deadend or find a problem, I recommend asking in the AWS Special Interest Group forum:
https://groups.google.com/forum/#!forum/kubernetes-sig-aws