I have created an AWS CloudWatch log group with a name which was not very future proof. Is it possible to rename such a log group? I could find no option to do this via the CloudWatch console.
I guess an alternative would be to export the content of the log group, create a new log group with the desired name, and then re-consume the exported logs to the new log group. I would prefer to avoid this approach though, as (apart from being tedious) this would reset the ingestionTime of all the log events in the log group.
It does not seem to be possible at the current time. The AWS CLI only lists the following log-group-related actions at this time:
create-log-group
delete-log-group
describe-log-groups
list-tags-log-group
tag-log-group
untag-log-group
My solution, since the log-group was only recently created and all the original log files were still on the EC2 instance, was to:
stop the CloudWatch Logs agent on the EC2 instance with $ /etc/init.d/awslogs stop.
ensure the application was not currently running/logging on the EC2 instance
delete the existing log-group on CloudWatch
move the existing log files out of the expected directory on the EC2 instance
edit the file /var/awslogs/etc/awslogs.conf on the EC2 instance to specify the desired log group name instead of the previous one.
start the CloudWatch Logs agent on the EC2 instance with $ /etc/init.d/awslogs start.
move each log file back into the expected directory on the EC2 instance, one at a time, from the earliest to the latest (in terms of file modification time), so the log agent will process all of them (and not just ignore files with non-latest modification time) and send them to the new log-group.
restart the application on the EC2 instance
Quite the tedious procedure, especially with having to manually move the log files, one at a time, in the correct order. Would be neat if there was functionality in the future to rename a log group.
A "solution" ...
Create the new log group.
Have the streams write to this log group.
Using aws cli get the events for each of the streams for the old log group and then put them to the new group.
I've got about 2,000 streams I'm about to do this too. We restructured are group names.
Related
Windows Logs: 1058,windows-computer-name.prod.domain.com
I see the above line in Cloudwatch Log Group
This is generated by Cloudwatch agent running on a Windows EC2
Question:
Is there a way (to change the Cloudwatch agent config file) to write the Windows computer name to the log group as a separate field on Cloudwatch log group?
(This would help me to query the separate windows computer name field to check if logs from a certain list of computer names are getting forwarded to the log group or not)
You can push custom metrics to CloudWatch Agent which will be accessible for further processing. You have to use StatsD agent for this purpose. How to push custom metrics to CW is shown below with StatsD example:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-custom-metrics-statsd.html
Looking into adding autoscaling of a portion of our application using AWS simple message queuing which would launch EC2 on-demand or spot instances based on queue backlog.
One question I had, is how do you deal with collecting logs from autoscaled instances? New instances are spun up based on an image, but then they are shut down when complete. Currently, if there is an issue with one of our services, which causes it to crash, we have a system to automatically restart the service, but the logs and core dump files are there to review. If we switch to an autoscaling system, where new instance are spun up, how do you get logs and core dump files when there is a failure? Particularly if the instance is spun down.
Good practice is to ship these logs and aggregate them somewhere else, and there are many services such as DataDog and Rapid7 which will do this for you at a cost.
AWS however provides CloudWatch logs, which gives you a central place to store and view logs. It also allows you then to give users access to logs on the AWS console without them having to ssh onto a server.
Shipping your logs to CloudWatch logs requires the installation of the CloudWatch agent on your server and specifying in the config which logs to ship.
You could install the CloudWatch agent once and create an AMI of that server to use in your autoscaling group, or install and configure the CloudWatch agent in userdata for every time a server is spun up.
All the information you need to get started can be found here:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html
I have multiple instances.
So i want to monitor server and application log for each instances on cloudwatch.
Can i use aws cloudwatch log on multiple instances?
Yes, you can do that.
Simply install the Cloudwatch log agent on each of the instance that you have and you are good to go
For instructions about installing and configuring Cloudwatch log agent click here
You can, but beware that if you pick the same log group / log stream for every instance you'll get comingled messages (which are painful to debug).
It's easy to change the configuration (I recommend using log streams named after the instance ID), but you have to do this when you provision the instance (which means that you actually need a provisioning step, rather than just spinning up an AMI).
I am trying out AWS Glue service to ETL some data from redshift to S3. Crawler runs successfully and creates the meta table in data catalog, however when I run the ETL job ( generated by AWS ) it fails after around 20 minutes saying "Resource unavailable".
I cannot see AWS glue logs or error logs created in Cloudwatch. When I try to view them it says "Log stream not found. The log stream jr_xxxxxxxxxx could not be found. Check if it was correctly created and retry."
I would appreciate it if you could provide any guidance to resolve this issue.
So basically, the job you add to Glue will either run if there's not too much traffic in the region your Glue is. If there are no resources available, you need to either manually re-add the job again or you can also bind yourself to events from CloudWatch via SNS.
Also, there are parameters you can pass to the job like maximunRetry and timeout.
If you have a Ressource not available, it won't trigger a retry because the job did not fail, it just didn't even started. But if you set the timeout to let's say 60 minutes, it will trigger an error after that time, decrement your retry pool and re-launch the job.
The closest thing I see to Glue documentation on this is here:
If you encounter errors in AWS Glue, use the following solutions to
help you find the source of the problems and fix them. Note The AWS
Glue GitHub repository contains additional troubleshooting guidance in
AWS Glue Frequently Asked Questions. Error: Resource Unavailable If
AWS Glue returns a resource unavailable message, you can view error
messages or logs to help you learn more about the issue. The following
tasks describe general methods for troubleshooting. • A custom DNS
configuration without reverse lookup can cause AWS Glue to fail. Check
your DNS configuration. If you are using Amazon Route 53 or Microsoft
Active Directory, make sure that there are forward and reverse
lookups. For more information, see Setting Up DNS in Your VPC (p. 23).
• For any connections and development endpoints that you use, check
that your cluster has not run out of elastic network interfaces.
I have recently struggled with Resource Unavailable thrown by Glue Job
Also i was not able to make a direct connection in Glue using RDS -it said "no suitable security group found"
I faced this issue while trying to connect with AWS RDS and Redshift
The problem was with the Security Group that the Redshift was using. There is a need to place a self referencing inbound rule in the Security Group.
For those who dont know what is self referencing inbound rule, follow the steps
1) Go to the Security Group you are using (VPC -> Security Group)
2) In the Inbound Rules select Edit Inbound Rules
3) Add a Rule
a) Type - All Traffic b) Protocol - All c) Port Range - ALL d) Source - custom and in space available write the initial of your security group and select it. e) Save it.
Its done !
if you were missing this condition in your Security Group Inbound Rules
Try creating the connection you will be able to create the connection.
Also job should work this time.
I have a subset of Windows EC2 instances that I would like to continuously copy files to whenever files are uploaded to a specific S3 bucket. Files will be uploaded to this bucket anywhere between once a month to several times a month but will need to be copied to the instances within an hour of upload. EC2 instances will be continually added and removed from this subset of instances. I would like this functionality to be controlled by the EC2 instance so that whenever a new instance is created, it can be configured to pull from this bucket. Ideally, this would be an instantaneous upon upload (vs a cron job running periodically). I have researched AWS Lamba and S3-notifications, and I am unsure if these are the correct methods to use. What solution is best suited to fit this model of copying files?
If you don't need "real time" presence of the files, you might think to run s3 sync on each instance by a cron job (easy one) or s3-notification->with some lambda works to deliver EC2 Run Command.
If the instances are in an autoscaling group, you can use aws s3 copy in the user data section of your launch config to accomplish this.