I'm having issues with Jenkins taking 15~ mins to build all our sites, whereas locally it takes roughly 1min per site. We have a Jenkins primary/secondary setup on Ec2 servers with the below configurations and they are using this for our CICD deployments.
Within Jenkins we are running a high intensive build job which builds and deploys multiple sites using Angular JS, we are building about 15 websites with all the build libraries and builders packages.
We used to have T-Family instance type that would take around 50~ mins to build these sites, we moved to C-Family instance type (c5a.4xlarge) and that reduced the build time to roughly 15~ mins but still, locally our MAC machines run these same jobs in less than 8~ mins compared to 15~ mins with AWS C-Family Instance Type.
Current Jenkins Instance config:
Name - Jenkins Primary
OS - Amazon Linux 2
Instance Type – c5a.4xlarge
vCPU- 16
Memory - 32 GB
Volume Type - gp3 SSD
IOPS - 6000
Throughput - 250 MBS
Name - Jenkins Secondary
OS - Amazon Linux 2
Instance Type – c5a.2xlarge
vCPU- 8
Memory - 16 GB
Volume Type - gp3 SSD
IOPS - 3000
Throughput - 300 MBS
I wanted to see what optimization options we have do to reduce this build time.
Thank you!!
Moved from this:
Name - Jenkins Primary
OS - Amazon Linux 2
Instance Type – c5a.4xlarge
vCPU- 16
Memory - 32 GB
Volume Type - gp3 SSD
IOPS - 6000
Throughput - 250 MBS
Name - Jenkins Secondary
OS - Amazon Linux 2
Instance Type – c5a.2xlarge
vCPU- 8
Memory - 16 GB
Volume Type - gp3 SSD
IOPS - 3000
Throughput - 300 MBS
To this:
Name - Jenkins Primary
OS - Amazon Linux 2
Instance Type – c5a.4xlarge
vCPU- 16
Memory - 32 GB
Volume Type - gp3 SSD
IOPS - 6000
Throughput - 250 MBS
Name - Jenkins Secondary
OS - Amazon Linux 2
Instance Type – c5a.2xlarge
vCPU- 8
Memory - 16 GB
Volume Type - gp3 SSD
IOPS - 3000
Throughput - 300 MBS
Related
We have recently migrated from Composer 1 to Composer 2. One of the task is heavily affected after this migration.
The task runs using BigqueryOperator. The query processes 50TB of data.
Composer 1 Configuration :
Web server machine type
composer-n1-webserver-2 (2 vCPU, 1.6 GB memory)
Cloud SQL machine type
db-n1-standard-2 (2 vCPU, 7.5 GB memory)
Worker nodes
Node count
3
Disk size (GB)
50
Machine type
e2-standard-4
Number of schedulers
1
The query use to take around 40 mins
Composer 2 Configuration:
Resources
Workloads configuration
Scheduler
4 vCPUs, 7.5 GB memory, 5 GB storage
Number of schedulers
2
Web server
2 vCPUs, 7.5 GB memory, 10 GB storage
Worker
4 vCPUs, 16 GB memory, 10 GB storage
Number of workers
Autoscaling between 4 and 8 workers
The same query takes around 1 hour 40 mins.
Does worker storage(Disk) reduction from 50GB(Composer 1)to 10 GB(Composer 2) is affecting the query run.
Does worker nodes play any role query computation or they just takes the tasks from queue and submit the query to Bigquery(in this case)?
This is a limitation as Composer 2 uses GKE Autopilot. Refer to this document for details of Composer 2.
Composer workers are intended to be used for launching and/or polling for the status of work launched elsewhere.
You can use ephemeral-storage limits on the KubernetesPodOperator e.g. k8s.V1ResourceRequirements(limits={"ephemeral-storage": "2Gi"})
Ephemeral volumes are designed for these use cases. Because volumes follow the Pod's lifetime and get created and deleted along with the Pod, Pods can be stopped and restarted without being limited to where some persistent volume is available.
I've only just starting using ecs and it seems like ebs is sky rocking.I wanna stay within the free tier without worrying about this My question is how do i lower ebs?
I'm using terraform ecs module. I changed the root_block_device_size to 10
module "ecs_cluster" {
cluster_instance_root_block_device_size = 10
}
This is the library I'm using Ecs module link(Terraform)
It mentions this
cluster_instance_root_block_device_size number
Description: The size in GB of the root block device on cluster instances.
Default: 30
cluster_instance_root_block_device_type string
Description: The type of the root block device on cluster instances ('standard', 'gp2', or 'io1').
Default: "standard"
For my ecs i have changed the block device size to 10 not sure if i should mess around with gp2 or io1 to lower it
Thanks!
Update using this https://aws.amazon.com/ebs/pricing/ and by
putting my configuration and it seems like gp3 and lowering the gigabytes does lower the price....However it seems like doing 10gb cannot be initialized with Terraform so i started deploying it with 30gb and lowered it to 10gb and it seemed to have worked...
That screenshot is for I/Os. Those are reads/writes to the disk volume(s). That's not related to the size of the volumes. Changing it from 30GB to 10GB is not going to impact the I/Os metric at all.
I/Os are only charged on magnetic EBS volume types. If you switched to an SSD based EBS volume type, like gp2, you would not be charged for the I/Os.
The AWS Free Tier includes "30 GB of Amazon EBS: any combination of General Purpose (SSD) or Magnetic". So you could have up to 30GB of free gp2 volumes, which would be much faster than the standard magnetic volumes you are using now, and you would also not be charged for I/Os on those volumes.
We have set up a dedicated cluster for our application on AWS.
This is the configuration of the cores ( we have 2 cores)
m5.xlarge
4 vCore, 16 GiB memory, EBS only storage
EBS Storage:64 GiB
Current dataset -
We are trying to run spark job which involves many joins and works with 80 million records
each record with 60 + fields
Issue we are facing -
When we trying to save the final dataframe as athena table, its taking more than 1 hour and timing out.
As we are the only one using the cluster, what should be our configuration to ensure that we use all the cluster resources optimally
Current configuration
Executor Memory : 2G
Dynamic Allocation Enabled : true
Number of Executor Cores : 1
Number of Executors : 8
spark.dynamicAllocation.executorIdleTimeout : 3600
spark.sql.broadcastTimeout : 36000
Looking at your config some observation -
You are using
m5.xlarge which is having 4 vCore, 16 GiB memory
Executor config
Number of Executor Cores : 1
Executor Memory : 2G
So at most 4 executor can spin up, and memory required by 4 executor is 8.
So at the end you are not utilizing all the resource.
Also as #Shadowtrooper said, save the data in partition (If possible in Parquet format) if you can, it will also save cost when you query in Athena.
We started sending a lot of data to a recently deployed application yesterday, that quickly used all the IOPS burst of the RDS instance that got stucked to 30 IOPS (The application was created on elastic beanstalk with a Postgresql database and 10 GB SSD storage)
Following the documentation I increased the storage space to 50 GB in order to get more IOPS (150 I guess). I did this 18 hours ago but the instance is still limited to 30 IOPS which create a very high latency on our application...
Any idea on how I can get this fixed?
Depending on the type of storage class you use, if you set it to apply immediately it should come into affect otherwise it will take affect in the next maintainance window.
I was trying to create t2.micro instance via AWS console.
I hope its free for one year, but it does not come with any instance storage.
So i wanted to add EBS Volume in this instance? is it free?
What is the maximum EBS volume i can add in t2.micro as free?
Model vCPU CPU Credits / hour
Mem (GiB) Storage
t2.micro 1 6 1 EBS-Only
30 GB of EBS is comes with the Free Tier. Go ahead use the 30 GB of EBS.
You can break it and use it either way want like 20 + 5 + 3 + 2 GB [ with magnetic or Genral Purpose SSD ].