I have just read Google's Cloud SQL's high availability documentation.
From what I understood in order for Google to:
Guarantee no data loss in case of primary node failure.
Allow clients to use standby node as read replica with strong consistency.
Google has to replicate writes in a synchronous way across multiple zones.
This seems like a very costly operation that should affect write transactions' latency. I however personally have not observed any significant latency differences between HA and non-HA version of GCP's Postgres.
How is it possible?
Not a definitive answer, but i hope it helps anyway.
The technology used here is the same that is used for regional persistent disks, which are highly optimized for these kind of multi-zone write scenarios. This basically means that the operation is costly but not as costly as you seem to expect. However even Google itself acknowledges the fact that there will be increased write latency. From a page directly linked from the documentation you shared:
Regional persistent disks are an option when write performance is less critical than data redundancy across multiple zones.
Documentation
You're confusing Read Replicas, HA and Legacy HA.
Cloud SQL some time ago was using Legacy HA which uses an explicit instance that replicates from the primary instance (in some way it is a read replica). In this particular case there could be replication lag because as you mention, the writes are synchronous and the performance could be impacted.
Legacy HA it is only available for MySQL.
In the case of Postgres it is expect you do not see a difference between HA and non-HA because in fact Postgres does not use this Legacy HA and uses the current HA schema which uses a single Regional Disk. non-HA also uses a single disk, the difference is that the disk is Zonal.
In other words, both (HA and non-HA) use a single disk, what changes is the scope of the disk (Regional or Zonal).
Finally since it is only a disk, there's no need to replicate like in a Read Replica or Legacy HA.
Related
can anyone elaborate on the difference between AWS Read Replica and readable Standby instances which AWS has offered recently?
I assume you're talking about the Readable Standby Instances available in Preview at the time of writing this.
Compared to the traditional read replicas, the main difference is the kind of replication involved. Replication to read replicas happens asynchronously. That means read replicas aren't necessarily up to date with the main database. This is something your workload needs to be able to deal with if you want to use that.
Readable standby instances on the other hand use synchronous replication. When you read from one of those instances your data will be up to date.
There are also a couple of other differences between the capabilities, but some things aren't finalised yet. The main difference is the kind of replication.
We're using AWS, and considering to use DynamoDB or Redis on our new service.
Below is our service's character
Insert/Delete occur over between hundreds and thousands per minute, and will be larger later.
We don't need quick search, only need to find a value with key
Data should not be lost.
There are another data that doesn't have a lot of Insert/Delete unlike 1.
I'm worried about when Redis server down.
When the Redis failure, our data will be removed.
That's why I'm considering to select Amazon DynamoDB.
Because DynamoDB is NoSQL, so Insert/Delete is so fast(slower than Redis, but we don't need to that much speed), and store data permanently.
But I'm not sure that my thinking is right or not.
If I'm thinking wrong or don't think another important point, I'm going appreciate when you guys teach me.
Thanks.
There are two type of Redis deployment in AWS ElastiCache service:
Standalone
Multi-AZ cluster
With standalone installation it is possible to turn on persistence for a Redis instance, so service can recover data after reboot. But in some cases, like underlying hardware degradation, AWS can migrate Redis to another instance and lose persistent log.
In Multi-AZ cluster installation it is not possible to enable persistence, only replication is occur. In case of failure it takes a time to promote replica to master state. Another way is to use master and slave endpoints in the application directly, which is complicated. In case of failure which cause a restart both Redis node at time it is possible to lose all data of the cluster configuration too.
So, in general, Redis doesn't provide high durability of the data, while gives you very good performance.
DynamoDB is highly available and durable storage of you data. Internally it replicates data into several availability zones, so it is highly available by default. It is also fully managed AWS service, so you don't need to care about Clusters, Nodes, Monitoring ... etc, which is considering as a right cloud way.
Dynamo DB is charging by R/W operation (on-demand or reserved capacity model) and amount of stored data. In may be really cheap for testing of the service, but much more expensive under the heavy load. You should carefully analyze you workload and calculate total service costs.
As for performance: DynamoDB is a SSD Database comparing to Redis in-memory store, but it is possible to use DAX - in-memory cache read replica for DynamoDB as accelerator on heavy load. So you won't be strictly limited with the DynamoDB performance.
Here is the link to DynamoDB pricing calculator which one of the most complicated part of service usage: https://aws.amazon.com/dynamodb/pricing/
i understand "spark.deploy.spreadOut" when set to true can benefit HDFS, but for S3 can setting to false have a benefit over true?
If you're running Hadoop and HDFS, it would not benefit you to use Spark Standalone scheduler for which that property applies. Rather, you should be running YARN, and the ResourceManager determines how executors are spread
If you are running Standalone scheduler in EC2, then setting that property will help, and the default is true.
In other words, where you're reading the data from is not the deciding factor here, the deploy mode for the master is
The better performance benefits would come from the number of files you're trying to read, and which formats you store the data in
This really depends on your workload.
If your S3 access is massive and is constrained by instance network IO,
setting spark.deploy.spreadOut=true will help, because it will spread it over more instances increasing the total network bandwidth available to the app.
But for the most workloads it will make no difference.
There is also cost consideration for "spark.deploy.spreadOut" parameter.
If your spark processing is large scale, you are likely using multiple AZs.
Default value "spark.deploy.spreadOut"= true will cause your workers to generate more network traffic on data shuffling, causing inter-AZ traffic.
Inter-AZ traffic on AWS can get costly
if the network traffic volume is high enough, you might want to cluster apps tighter by spark.deploy.spreadOut"= false, instead of spreading them because of the cost issue.
Single-Region Spanner is advertised with a 99.99% availability SLA. In the US-based configuration, there will be exactly three replicas per node, all in Council Bluffs, Iowa. Can you share information that breaks down why the 99.99% (~one hour of downtime per year) is believable, especially in the case of geographically-local disasters? I assume that Google has done a thorough analysis, or else it would not advertise the SLA, but I cannot find a detailed paper.
In the event of a regional failure, what recovery procedures will Google carry out and with what recovery time / expected data loss?
(I understand that multi-region may be available, and have seen some pricing data, but will not discuss this here).
Spanner automatically replicates data for high availability. As you stated, regional instances have three full copies of data. The key is that they are replicated across three zones within the region which have independent power, cooling, networking, etc. Zones generally fail independently for each other, so your other replicas can continue serving reads and writes even if one zone goes down. Multi-region provides even greater availability by replicating across regions.
Zonal failures are very rare and would be transparent to your application; Cloud Spanner automatically reroutes requests to replicas that are able to serve the request. It would be even rarer for a region to go down with data loss. Google takes many measures against disasters.
Further out we will expose managed backups, but these would still be stored within Google data centers. We're also working on a Dataflow connector to help you import/export data should you want to manage your own backups.
I need to scale the Aurora DB up and then back down at later sometime but I dont need any downtime not even a bit. I though of doing it by creating aurora replica and promoting it then scaling primary and promoting back to primary. But this involve downtime. Please suggest any alternative way.
i would first ask you in what sense do you need to scale up? writing? reading? if you currently run a server with a high-read/low-write ratio, i'd suggest another read-replica (or more) and then configure your application specifically to use the read replica in those specific cases where you're only making queries and not modifying the data. in this sense, you're offloading your writes to a separate instance of the same data and letting the primary deal with mostly writes. if you did this, i'd also suggest spreading out the read replicas across other availability zones, in case the AZ of your primary goes down, another read-replica will be auto-promoted to primary.
armed with that info, i would suggest you scale up with read replicas that your application is configured to use, then you can bring down those replicas as you scale down without ever bringing down the Aurora primary. it's more of a configuration problem in dealing with multiple RDS endpoints, essentially.
HTH.
one caveat is that there is some delay in replication between primary and read replicas (my instances show about 10-20 milliseconds), so you have to keep this in mind if you perform a write then a read in fast succession -- if your first read after the initial write happens 'too fast', the replica might not see the change and may either see no data (if you're creating) or stale data (if you're updating).
in general, this isn't an issue until you're under some heavy load and the writes on the primary are backed up and you start reading from the read replica before the writes have been applied to the primary.
HTH.