I have installed Mariadb-10.6.8 on Amazon Web Service (AWS) RDS instance. The RDS instance has a disk capacity of 150GB and 16GB RAM and it hosts 3 databases of total size 13gb. This database serves a website which hardly has much DML operations and predominately read data from this database using stored procedures. These stored procs extensively use temporary tables and the query performance is well under 1 sec. On the website there would be only around 10 to 25 concurrent users most of the time and at peak time there would be 30 to 35 users.
The problem
When I start the RDS instance the disk space available is 137 GB (with 13 gb used by the data held by the databases). Now as the day progresses and the users access the website the disk space starts reducing drastically and reduces by 35gb in 1 day (though there are hardly couple of inserts/updates). If now I restart the RDS instance then the disk space of 137GB is available again and as the day progresses the disk space keeps on reducing again. So the issue is why is the disk space reducing automatically.
Related
I have a web application that is caching data in CSV files, and then in response to an HTTP request, reading the CSV files into memory, constructing a JavaScript Object, then sending that Object as JSON to the client.
When I run this on my Local Server on my Macbook Pro (2022, Chip: Apple M1 Pro, 16GB Memory, 500GB Hard Drive), the 24 CSV files at about 15MB each are all read in about 2.5 seconds, and then the subsequent processing takes another 3 seconds, for a total execution time of about 5.5 seconds.
When I deploy this application to AWS, however, I am struggling to create a comparably performant environment.
I am using AWS Elastic Beanstalk to spin up an EC2 instance, and then attaching an EBS volume to store the CSV files. I know that as the EBS is running on a separate instance, there is possible network latency, but my understanding is typically pretty negligible as far as overall effect on performance.
What I have tried thus far:
Using a Compute focused instance (c5.4xlarge) which is automatically EBS optimized. Then using a Provisioned IOPS (io2) with 1000 GiB storage and 400 IOPS. (Performance, about 10 seconds total)
Using a High Throughput EBS volume, which is supposed to offer greater performance for sequential read jobs (like what I imagined reading a CSV file would be), but that actually performed a little worse than the Provisioned IOPS EBS instance. (Performance, about 11 seconds total)
Can anyone offer any recommendations for which EC2 Instance and EBS Volume should be configured to achieve comparable performance with my local machine? I don't expect to get it matching exactly, but do expect that it can be closer than about twice as slow as the local server.
How can I load balance my Relational Database on AWS so that I don't have to pay for a large server that I don't need 99% of the time? I am pretty new to AWS so I am not totally sure if this is even possible. My app experiences surges (push notifications) that would crash the smaller DB instance class, but the larger (r5.4xlarge) isn't needed 99% of the time. How can I avoid paying for the larger instance? We are using MySQL.
This is the max CPUs utilization over the past 2 weeks for 16 CPUs and 128 GiBs RAM
I'm just setting up a hight availability WordPress network and I need to decide how much RAM I need for the database instance. On a web server you run "top" and find out how much RAM is being used per MySql process and then look in your config file and look at the maximum number of processes that are allowed to run.
How do you calculate how much RAM you will need on a High availability MySQL database running in AWS LightSail? The plans seem very light on RAM. For example a $20 webserver gets 4GB of RAM whereas a $60 database server get 1GB of RAM. Why is this and how many processes will 1GB run?
We started sending a lot of data to a recently deployed application yesterday, that quickly used all the IOPS burst of the RDS instance that got stucked to 30 IOPS (The application was created on elastic beanstalk with a Postgresql database and 10 GB SSD storage)
Following the documentation I increased the storage space to 50 GB in order to get more IOPS (150 I guess). I did this 18 hours ago but the instance is still limited to 30 IOPS which create a very high latency on our application...
Any idea on how I can get this fixed?
Depending on the type of storage class you use, if you set it to apply immediately it should come into affect otherwise it will take affect in the next maintainance window.
Created a new Ubuntu c3.xlarge instance and when I get to storage options I get the option to change ROOT to General Purpose SSD, Provisioned IOPS or magnetic, also if I pick Provisioned IOPS i can set another value. Additional data storage under Instance Store 0 has no options but if change to EBS then I have the same options.
I'm really struggling to understand:
The speed of each option
the costs of each option
The Amazon documentation is very unclear
I'm using this instance to transfer data from text files into a Postgres relational database, these files have to be processed line by line with a number of INSERT statements per line so is slow on my local computer (5 million rows of data takes 15 hours). Originally the database was separately on RDS but it was incredibly slow, so I installed the database locally on the instance itself remove network latency which has speed up things a bit but it is still considerably slower than my local humble linux server.
Looking at the instance logs whilst loading the data CPU instance is only at 6% so now thinking that disk may be limiting the factor. The database will be using the / (Not sure if SSD or magnetic - how can I find out) disk and the data files are on the /mnt (using Instance Store 0) disk.
I only need this instance to do two things:
Load database from datafiles
Create Lucene Search Index from database
(so the database is just an interim step)
The Search Index is transferred to an EBean Server and then I don;t need this instance for another month when I then repeat the process with new data so with that in mind I can afford to spend more money for faster processing because I'm only going to use 1 day a month, then I can stop the instance and incur no further costs ?
Please what can I do to determine the problem and speed things up ?
Here is my personal guideline:
If the volume is small (<33G) and only require a eventual burst in performance, such as a boot volume, use magnetic drives.
If you need predictable performance and high throughput, use PIOPS volumes and EBS optimized instances.
Otherwise, use General Purpose SSD.
Your CPU is only at 6%, maybe you can try to use multi-process?
Did you test your remote instance's volume's I/O performance?
PIOPS is expensive, but it did not significantly better than gp2, the only advantage is stable.
For example, I create a 500G gp2 and a 500G PIOPS with 1500IOPS, then I try to insert and find 1,000,000 documents by mongodb, then I check the io performanace by such as mongoperf/iostat/mongostat/dstat
Each volume's iops performance is expect to 1500,
but gp2's iops is unstable, almost from 700 to 1600(r+w), if only read, it can brust to 4000, if only write, it just reach 800.
piops is perfect stable, it iops is almost 1470.
To your situation, I suggest to consider about gp2 (volume size depend on your iops demand, 500G gp2 = 1500iops, 1T gp2 = 3000iops(maximum))