I have an Elastic Kubernetes Cluster(EKS) running in AWS , In the cluster many services and pods are running .I want to use AppDynamics to monitor the services and pods . I am new to AppDynamics so I don't know much about it . but i am confused in some areas
What are the performance metrics(CPU usages , no of instances... ) should I use for monitor the
cluster
How can I monitor the cluster , how to setup AWS with AppDynamics to monitor everything
The Cluster Agent is used for monitoring AWS EKS, additionally the Cluster Agent Operator can be used to setup additional Infra / Network monitoring.
Compatibility: https://docs.appdynamics.com/21.9/en/infrastructure-visibility/monitor-kubernetes-with-the-cluster-agent/cluster-agent-requirements-and-supported-environments
Install (Cluster Agent): https://docs.appdynamics.com/21.9/en/infrastructure-visibility/monitor-kubernetes-with-the-cluster-agent/install-the-cluster-agent
(You will need to grab / build an image and then install using Kubernetes CLI or the Cluster Agent Helm Chart)
Install (Infra Agent / Network Visibility - requires the Cluster Agent): https://docs.appdynamics.com/21.9/en/infrastructure-visibility/monitor-kubernetes-with-the-cluster-agent/install-the-cluster-agent/install-infrastructure-visibility-with-the-cluster-agent-operator
Metrics: https://docs.appdynamics.com/21.9/en/infrastructure-visibility/monitor-kubernetes-with-the-cluster-agent/use-the-cluster-agent/monitor-cluster-health
As to what Metrics to actively monitor this is a bit subjective, however there are plenty of guides around to help, e.g:
https://www.kubermatic.com/blog/the-complete-guide-to-kubernetes-metrics/
https://sematext.com/blog/kubernetes-metrics/
Related
We host Docker containers on AWS infrastructure using AWS EKS. My reading so far shows that the kubectl command-line tool gives me commands to query and manipulate the EKS cluster. The aws eks command-line tool also gives me commands to do this. To my inexperienced eye, they look like they offer the same facilities.
Are there certain situations when it's better to use one or the other?
aws eks command is for interacting with AWS EKS proprietary APIs to perform administrative tasks such as creating cluster, updating kubeconfig with correct credentials etc.
kubectl is an open source ClI tool which let you interact with kubernetes API server to perform tasks such create pods, deployments etc.
You can not use aws eks command to interact with Kubernetes API Server and perform any kubernetes specific operations because it does not understand kubernetes APIs.
Similarly you can not use kubectl to interact with AWS EKS proprietary APIs because kubectl does not understand it.
I am trying to build on demand AWS jmeter(can be any testing tool like SOAP UI, Selenium ) instance to using Jenkins. Not looking for Server client Jmeter distribution architecture.
This is to provide cost effective solution to the spawn on demand jmeter(Not containerization )instance using Jenkins. New instance need JNLP or jenkins agent to establish connectivity with Jenkins Master.
Can some one provide me any documentation and codes(CLI) to spin up aws instance with or without AMI ?
You can use AWS CLI to manage instances (create, launch, shut down, terminate, etc.)
Example command would be:
aws ec2 run-instances --image-id your_image_id --count how_many_instances_you_want --instance-type desired_EC2_instance_type --key-name your_key_pair --security-groups your_EC2_security_group_name
Make sure that the security group allows the following ports:
the port you define as server_port, by default 1099
the port you define as server.rmi.localport
the port(s) you define as client.rmi.localport
More information:
Remote hosts and RMI configuration
Apache JMeter Properties Customization Guide
Am not sure if your are looking for this kind of setup.
Use terraform, infra as code. You will be able to spawn all the resources that are required for your test. The steps will follow like this,
Create a jmeter Docker image
Push it to ECR
Create a Cluster in ECS
Create a Task definition
Create a service in ECS cluster where it uses the Jmeter image and you can use fargate serverless.
On all the above you can use Jenkins CI/CD where you can trigger you terraform code.
This happens while trying to create a VPC-native GKE cluster. Per the documentation here the command to do this is
gcloud container clusters create [CLUSTER_NAME] --enable-ip-alias
However this command, gives below error.
ERROR: (gcloud.container.clusters.create) Only alpha clusters (--enable_kubernetes_alpha) can use --enable-ip-alias
The command does work when option --enable_kubernetes_alpha is added. But gives another message.
This will create a cluster with all Kubernetes Alpha features enabled.
- This cluster will not be covered by the Container Engine SLA and
should not be used for production workloads.
- You will not be able to upgrade the master or nodes.
- The cluster will be deleted after 30 days.
Edit: The test was done in zone asia-south1-c
My questions are:
Is VPC-Native cluster production ready?
If yes, what is the correct way to create a production ready cluster?
If VPC-Native cluster is not production ready, what is the way to connect privately from a GKE cluster to another GCP service (like Cloud SQL)?
Your command seems correct. Seems like something is going wrong during the creation of your cluster on your project. Are you using any other flags than the command you posted?
When I set my Google cloud shell to region europe-west1
The cluster deploys error free and 1.11.6-gke.2(default) is what it uses.
You could try to manually create the cluster using the GUI instead of gcloud command. While creating the cluster, check the “Enable VPC-native (using alias ip)” feature. Try using a newest non-alpha version of GKE if some are showing up for you.
Public documentation you posted on GKE IP-aliasing and the GKE projects.locations.clusters API shows this to be in GA. All signs point this to be production ready. For whatever it’s worth, the feature has been posted last May In Google Cloud blog.
What you can try is to update your version of Google Cloud SDK. This will bring everything up to the latest release and remove alpha messages for features that are in GA right now.
$ gcloud components update
I came across an open source Kubernetes project KOPS and AWS Kubernetes service EKS. Both these products allow installation of a Kubernetes cluster. However, I wonder why one would pick EKS over KOPS or vice versa if one has not run any of them earlier.
This question does not ask which one is better, but rather asks for a comparison.
The two are largely the same, at the time of writing, the following are the differences I'm aware of between the 2 offerings
EKS:
Fully managed control plane from AWS - you have no control over the masters
AWS native authentication IAM authentication with the cluster
VPC level networking for pods meaning you can use things like security groups at the cluster/pod level
kops:
Support for more Kubernetes features, such as API server options
Auto provisioned nodes use the built in kops node_up tool
More flexibility over Kubernetes versions, EKS only has a few versions available right now
Other significant difference is that EKS is an AWS product so you require an AWS account but kops allows to run Kubernetes in AWS but also in GCE and DigitalOcean.
My goal is to understand how can I auto-scale a Hadoop cluster on AWS EC2.
I am exploring AWS offerings from elastic scaling perspective for a Hadoop as service (EMR) and Hadoop on EC2.
For EMR, I gathered that using CloudWatch, performance metrics can be monitored and the user can be alerted once they reach the set threshold, thereafter the cluster can be scaled up or down depending on its utilization state.
This approach would require some custom implementation to automate the steps.(correct me if I am missing anything here)
For Hadoop on EC2, I came across with the auto scaling option which can add or remove instances as per configured scaling policies.
But I am not clear how a newly added node would get bootstrapped to the cluster automatically? How would YARN know that it can spawn a new container on this newly added node?
Does auto-scaling work for master-slave kind of setup as well or is limited to the web application?
There is 'Qubole' offering services to manage Hadoop on AWS as well....should that be used for automatically managing scaling the cluster?