I found this excellent service https://app.cloudzy.io/ to backup/create snapshots for my AWS instances on a schedule and being able to set the retention. Very simple and easy to use
I just got an email from them that they are shutting down the service.
Now I'm looking for something similar that is affordable. Any recommendations?
A quick google search found exactly what you needed.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/TakeScheduledSnapshot.html
It’s fairly easy to roll your own with a scheduled Lambda function. I wrote a script similar to the one here: https://serverlesscode.com/post/lambda-schedule-ebs-snapshot-backups/
If you are willing to spend money, N2WS is your next best bet. https://n2ws.com/
Related
What is the recommended approach to store the logs of applications deployed on Kubernetes? I read about ELK stack, but not sure about the pros and cons. Needs recommendations.
If you ask specifically about storing application logs in kubernetes cluster, there are a few different approaches. First I would recommend you to familiarize with this article in the official kubernetes documentation.
As per my experience with the Kubernetes logging, I would suggest you go with EFK stack (Fluentd/flunetbit --> Kafka --> Logstash/flunetd --> Elasticserach --> kibana), this one has initial challenges during setup but once this is up and running, it will be like a super scalable system where you don't need to worry about volume of logs you are shipping.
Another approach you can take is shipping logs directly from fluentd/fluentbit/filebeat to Elasticsearch. The drawback of this approach is if ES has some issue then you may lose your logs.
I hope it helps.
I want to emphasize the response from #javajon. There is a KataCoda exercise specifically for logging at https://katacoda.com/javajon/courses/kubernetes-observability/efk.
Logging is a very large topic with lots of variables. In order to get any specific advice, you'll need to comment about your goals for logging. Is it related to performance, compliance, security, debugging, observability or something else?
Try to get some knowledge by yourself.
Every storage have some pros and cons according to requirement we use them.
Visit https://medium.com/volterra-io/kubernetes-storage-performance-comparison-9e993cb27271
and learn more.
I will surely somehow help.
I am trying to use DynamoDB for the backend DB of my application, but am having a hard time finding useful information associated to it.
What is the best source of examples and tutorial information for syntax structures etc?
AWS docs are really confusing. Or am I the only person sitting with these problems?
Oh and is the newly launched AWS DocumentDB (Basically MongaDB) going to make DynamoDB pointless to learn, or is there still merit in learning DynamoDB?
The pricing model between DocumentDB and DynamoDB are completely different - there is definitely a place for both - imo, dynamodb is not going away any time soon.
As far as tutorials - there are tons of AWS reinvent videos on youtube, and this site allows you to search/find them easily: https://reinventvideos.com/. Good place to start.
We want to setup cloudwatch in more that 50 servers for which in general we will have to do it manually logging into each server.But we would like to reduce the manual work.
While browsing through we found below two ideas:
1)Opswork( aws internally uses chef)
2) Chef
Are the above approaches correct to achieve what intend to?
Which approach is best suitable?
Your suggestions will be of great help... Thank you
We performed this activity using chef.The process was simple.
There are a number of cookbooks already available on the chef supermarket which is of great help to beginners.
We did not try Opswork so i will not be able to comment which is a better approach.
I've been a mobile developer for a few years, and Im looking to expand to cloud integration with my apps. Im looking into AWS solutions to fill this need. I don't know a ton about servers or cloud capabilities, so I'm trying to get pointed in the right direction, and maybe be introduced to some good resources.
My goal is to be able to upload some images to AWS and manipulate these images in the cloud. I'm sure that I'll need S3 to store my images, but is an EC2 instance the correct thing to use to perform the manipulation? This is where my lack of knowledge of servers is holding me back.
I think that the best answer I could get would be a comment on whether my needs from AWS are what I listed above, and a point in the right direction towards articles to tutorials of how to get things up and running.
Thanks much for the help!
What I ended up doing was using AWS Lambda to accomplish what I needed. Running a node.js based lambda function with ffmpeg-like manipulation on the images/media that I was uploading worked out quite well.
Side note :: The processing that I was doing was fairly lightweight, so it worked well with lambda. If things scale up any further I might consider switching the processing to an EC2 instance.
We're trying to move to AWS and to use DynamoDB. It'd be nice to keep everything under DynamoDB so there aren't extraneous types of databases, but aside from half complete research projects I'm not really finding anything to use for a scheduler. There's going to be dynamically set schedules in the range of thousands+, possibly with many running at the same time. For languages, Java or at least JVM would be awesome.
Does anyone know a good Scheduler for DynamoDB or other AWS technology?
---Addendum
When I say scheduler I'm thinking of something all purpose like quartz. I want to set a cron and it runs at that time with the code I give it. This isn't doing some AWS task, this is a task internal to our product. SWF's cron runs inside the VM, so I'm worried what happens when the VM is down. Data Pipeline seems a bit too much. I've been looking into making a dynamodb job store for quartz, consistent read might get around the transaction and consistency issues, but I'm hesitant, might be biting off a lot with a lot of hard to notice problems.
Have you looked at AWS Simple Workflow? You would use the AWS Flow Framework to program against the service, and they have a well documented Java API with lots of samples. They support continuous workflows with timers which you can use to run periodic code (see code example here). I'm using SWF and the Flow Framework for Ruby to run async code that gets kicked off from my main app, and it's been working great.
Another new option for you is to look at AWS Lambda. You can attach your Lambda function code directly to a DynamoDB table update event, and Lambda will spin up and shut down the compute resources for you, without you having to manage a server to run your code. Also, recently, AWS launched the ability to call the Lambda function directly -- e.g. you could have an external timer or other code that triggers the function on a specific schedule.
Lastly, this SO thread may have other options for you to consider.
Another option is to use AWS Lambda Scheduled Functions (newly announced on October 8th 2015 at AWS re:Invent).
Here is a relevant snippet from the blog (source):
Scheduled Functions (Cron)
You can now invoke a Lambda function on a regular, scheduled basis. You can specify a fixed rate (number of minutes, hours, or days between invocations) or you can specify a Cron-like expression: