I have a k8s cluster deployed in AWS's EKS and I want to change horizontal-pod-autoscaler-sync-period from the default 30s value.
How can I change this flag?
Unfortunately you are not able do this on GKE, EKS and other managed clusters.
In order to change/add flags in kube-controller-manager - you should have access to your /etc/kubernetes/manifests/ dir on master node and be able to modify parameters in /etc/kubernetes/manifests/kube-controller-manager.yaml.
GKE, EKS and similar clusters manages only by their providers without getting you permissions to have access to master nodes.
Similar questions:
1) horizontal-autoscaler-in-a-gke-cluster
2) change-the-horizontal-pod-autoscaler-sync-period-with-gke
As a workaround you can create cluster using kubeadm init and configure/change it in any way you want.
Related
I am trying to deploy a Batch Compute Environment in a heavily restricted AWS environment. For billing purposes, all resources created need to be tagged (e.g. billTo: billId), and if I try to create a resource without this tag I am blocked by an explicit deny. When the Batch Compute Environment tries to create an ECS Cluster, I get the following error because it does not pass tags to it.
User: arn:aws:sts::<accountId>:assumed-role/<roleName> is not authorized to perform ecs:CreateCluster on resource: * with an explicit deny
There are two places to specify tags when creating a Batch Compute Environment (tag the compute environment and tag the EC2 resources used by the compute environment). I tried adding the billTo tag in both places but still hit the same error.
Does anyone know if it is possible to get Batch to tag the ECS Cluster it tries to create when making a new Batch Compute Environment?
Note: I also tried figuring out how to pass an existing ECS Cluster, but this is not possible (How to Set an existing ECS cluster to a compute environment in AWS Batch)
As of 2021-07-15, AWS does not provide a way to tag the ECS Cluster automatically made by the Batch Compute Environment. The solution is to
Get in contact with the system administrators
Have them lift the SCP (service control policy) causing the explicit deny for a short window
Create the Batch Compute Environment and add the tag to the ECS Cluster
Have the system administrators put the SCP back in place
Hope you don't have to redeploy and bug the system administrators again
Hopefully AWS will fix this issue and allow the ECS Cluster to be tagged by the Batch Compute Environment.
I've faced with the problem while using AWS SDK. Currently I am using SDK for golang, but solutions from other languages are welcome too!
I have ECS cluster created via SDK
Now I need to add EC2 containers for this cluster. My problem is that I can't use Amazon ECS Agent to specify cluster name via config:
#!/bin/bash
echo ECS_CLUSTER=your_cluster_name >> /etc/ecs/ecs.config
or something like that. I can use only SDK.
I found method called RegisterContainerInstance.
But it has note:
This action is only used by the Amazon ECS agent, and it is not
intended for use outside of the agent.
It doesn't look like working solution.
I need to understand how (if it's possible) to create working ECS clusterusing SDK only.
UPDATE:
My main target is that I need to start specified count of servers from my Docker image.
While I am investigating this task i've found that I need:
create ECS cluster
assign to it needed count of ec2 instances.
create Task with my Docker image.
run it on cluster manually or as service.
So I:
Created new cluster via CreateCluster method with name "test-cluster".
Created new task via RegisterTaskDefinition
Created new EC2 instance with ecsInstanceRole role with ecs-optimized AMI type, that is correct for my region.
And there place where problems had started.
Actual result: All new ec2 instances had attached to "default" cluster (AWS created it and attach instance to it).
If I am using ECS agent I can specify cluster name by using ECS_CLUSTER config env. But I am developing tool that use only SDK (without any ability of using ECS agent).
With RegisterTaskDefinition I haven't any possibility to specify cluster, so my question, how I can assign new EC2 instance exactly to specified cluster?
When I had tried to just start my task via RunTask method (with hoping that AWS somehow create instances for me or something like that) I receive an error:
InvalidParameterException: No Container Instances were found in your cluster.
I actually can't sort out which question you are asking. Do you need to add containers to the cluster, or add instances to the cluster? Those are very different.
Add instances to the cluster
This is not done with the ECS API, it is done with the EC2 API by creating EC2 instances with the correct ecsInstanceRole. See the Launching an Amazon ECS Container Instance documentation for more information.
Add containers to the cluster
This is done be defining a task definition, then running those tasks manually or as services. See the Amazon ECS Task Definitions for more information.
We need to access the kubelet logs on our Kubernetes node (which is in AWS) to investigate an issue we are facing regarding Kubernetes error (see Even after adding additional Kubernetes node, I see new node unused while getting error "No nodes are available that match all of the predicates:).
Kubectl logs only gets logs from pod. To get kubelet logs, we need to ssh into the k8s node box - (AWS EC2 box). While doing so we are getting error "Permission denied (publickey)" which means we need to set the ssh public key as we may not be having access to what were set earlier.
Question is if we set the new keys using kops as described in https://github.com/kubernetes/kops/blob/master/docs/security.md, would we end up creating any harm to existing cluster? Would any of the existing services/access stop working? Or would this only impact manual ssh to the AWS EC2 machines?
You would need to update the kops cluster using kops cluster update first. However, this would not change the SSH key on any running nodes.
By modifying a cluster using kops cluster update you are simply modifying the Launch Configurations for the cluster. This will only take effect when new nodes are provisioned.
In order to rectify this, you'll need to cycle your infrastructure. The only way to do this is to delete the nodes and control plane nodes one by one from the ASG.
Once you delete a node from the ASG, it will be replaced by the new launch configuration with the new SSH key.
Before you delete a node from AWS, you should drain it it first using kubectl drain:
kubectl drain <nodename> --ignore-daemonsets --force
According to documentation of both kops and aws, the dedicated kops user needs IAMFullAccess permission to operate properly.
Why is this permission needed?
Is there a way to avoid (i.e. restrict) this, given that it is a bit too intrusive to create a user with such a permission?
edit: one could assume that the specific permission is needed to attach the respective roles to the master(s) and node(s) instances;
therefore perhaps the question / challenge becomes how to:
not use IAMFullAccess
sync with the node creation / bootstrapping process and attach the above roles; (perhaps create a cluster on pre-configured instances? - no idea if kops provides for that)
As far as I understand kops design, it's meant to be end to end tool for provisioning you with k8s clusters. If you want to provision your nodes separately and deploy k8s on them I would suggest to use other tool, such as kubespray or kubeadm:
https://github.com/kubernetes-incubator/kubespray
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
We have created a single-master three-node worker cluster on AWS using Terraform, user-data YAML files, and CoreOS AMIs. The cluster works as expected but we are now in need to scale the master's up from one to three for redundancy purposes. My question is: other than using etcd clustering and/or the information provided on http://kubernetes.io/docs/admin/high-availability/, do we have any options to deploy a new or scale-up the existing cluster with multi-master nodes? Let me know if more details are required to answer this question.
The kops project can set up a high-availability master for you when creating a cluster.
Pass the following when you create the cluster (replacing the zones with whatever is relevant to you):
--master-zones=us-east-1b,us-east-1c,us-east-1d
Additionally, it can export Terraform files if you want to continue to use Terraform.