I have enabled this option.
Problem is:
If I don't press snapshot log button log, is not going to s3.
Is there any method through which log publish to s3 each day?
Or how log file rotation option is working ?
If you are using default instance profile with Elastic Beanstalk, then AWS automatically creates permission to rotate the logs to S3.
If you are using custom instance profile, you have to grant Elastic Beanstalk permission to rotate logs to Amazon S3.
The logs are rotated every 15 minutes.
AWS Elastic Beanstalk: Working with Logs
For a more robust mechanism to push your logs to S3 from any EC2 server instance, you can pair LogRotate with S3. I've put all the details in this post as a reference whicould should be able to achieve exactly what you're describing.
Hope that helps.
NOTICE: if you want to rotate custom log files, then, depending on your container, you need to add links to your custom log files in a proper places. For example, consider Ruby on Rails deployment, if you want to store custom information, eg. some monitoring using Oink gem in oink.log file, add proper link in /var/app/support/logs using .ebextensions
.ebextensions/XXXlog.config
files:
"/var/app/support/logs/oink.log" :
mode: "120400"
content: "/var/app/current/log/oink.log"
This, after deploy, will create symlink:
/var/app/support/logs/oink.log -> /var/app/current/log/oink.log
I'm not sure why permissions 120400 are used, I took it from the example in Amazon AWS doc page http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html (seems like 120xxx is for symlinks in unix fs)
This log file rotation is good for archival purpose, but difficult to search and consolidate when you need the most.
Consider using services like splunk or loggly.
Related
I'm an application developer with very limited knowledge of infrastructure. At my last job we frequently deployed Java web services (built as WAR files) to Elastic Beanstalk, and much of the infrastructure had already been set up before I ever started there, so I got to focus primarily on the code and not how things were tied together. One feature of Elastic Beanstalk that often came in handy was the button to "Request Logs," where you can select either the "Last 100 Lines" or the "Full Logs." What I'm used to seeing when clicking this button is to directly view the logs generated by my web service.
Now, at the new job, the infrastructure requirements are a little different, as we have to Dockerize everything before deploying it. I've been trying to stand up a Spring Boot web app inside a Docker container in Elastic Beanstalk, and have been running into trouble with that. And I also noticed a bizarre difference in behavior when I went to "Request Logs." Now when I choose one of those options, instead of dropping me into the relevant log file directly, it downloads a ZIP file containing the entire /var/log directory, with quite a number of disparate and irrelevant log files in there. I understand that there's no way for Amazon to know, necessarily, that I don't care about X log file but do care about Y log file, but was surprised that the behavior is different from what I was used to. I'm assuming this means the EB configuration at the last job was set up in a specific way to filter the one relevant log file, or something like that.
Is there some way to configure an Elastic Beanstalk application to only return one specific log file when you "Request Logs," rather than a ZIP file of the /var/log directory? Is this done with ebextensions or something like that? How can I do this?
Not too sure about the Beanstalk console, but using the EBCLI, if you enable CloudWatch log streaming (note that this would cost you to store logs in CloudWatch) for your Beanstalk instances, you can perform:
eb logs --stream --log-group <CloudWatch logGroup name>
The above command basically gives you the logs for your instance specific to the file/log group you specified. In order for the above command to work, you need to enable CloudWatch log streaming:
eb logs --stream enable
As an aside, to determine which log groups your environment presently has, perform:
aws logs describe-log-groups --region <region> | grep <beanstalk environment name>
Is there any way to stream/push docker app logs to S3 bucket?
I know following 2 ways
Configure cloud watch logs/stream - All logs (both info & Error logs) are getting merged in this approach
Configure graylogs2 to push every log message and collect and then push to S3 bucket - Need to maintain graylogs2 app.
I am looking for any easy way to push docker app/error logs S3 Bucket
Thanks
A possible solution, though it's hard to tell for your case, is to run logstash in a separate container, and have your app direct logs to logstash. Since Logstash’s logging framework is based on Log4j 2 framework, it will likely be familiar to you. A plugin already exists for logstash to push to S3 on your behalf.
You can configure your existing log4j2 to emit to a port that logstash is running on.
If even this is considered too much maintenance for you, your best solution is probably just setting up a cron to run rsync.
I am using Amazon EB for the first time. I've setup a Rails app running on linux and puma.
So far, I've been viewing logs through the eb logs command. I know that we can set EB to rotate the logs to S3 or stream it to CloudWatch.
My question here revolved around the deletion of the various log files.
Will the various logs, such as puma.log be deleted automatically or must I do it myself?
If i setup log rotations to S3, will the log files on the EC2 instance be deleted (and a fresh copy created in its place) when it gets rotated to S3? Or does it just keep growing indefinitely?
If i stream it to CloudWatch, will the same copy of the log be kept on the EC2 instance and grow indefinitely?
I've googled around but can't seem to find any notion of "Log management" or "log deletion" in the docs or on SO.
I'm using beanstalk on a LAMP project and I can answer a few of your questions.
You have to setup your log rotation policy at least on your app logs. Check if your base image already rotate this logs for you. The config should be in /etc/logrotate.conf for linux
When you use S3 logs with Beanstalk, it already tails and delete the logs after 15min. http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html#health-logs-s3location
The same copy of the log will be kept in your EC2 instance. Your log rotation policy /etc/logrotate.conf will be the one that will delete it. awslogs will keep some metadata to know which was the processed chunk of the logs so it does not create duplicates.
If you want an example on how to use cloudwatch logs with elasticbeanstalk check: http://www.albertsola.pro/store-aws-beanstalk-symfony-and-apache-logs-in-cloudwatch-logs/
I realized that the default logrotation for AWS EB's docker is 5 files of 10M each. That's not enough for me.
I found the config at /etc/logrotate.elasticbeanstalk.hourly/logrotate.elasticbeanstalk.applogs.conf and it reads as follows:
$ cat /etc/logrotate.elasticbeanstalk.hourly/logrotate.elasticbeanstalk.applogs.conf
/var/log/eb-docker/containers/eb-current-app/* {
size 10M
rotate 5
missingok
compress
notifempty
copytruncate
dateext
dateformat %s
olddir /var/log/eb-docker/containers/eb-current-app/rotated
}
I'd like to change that to a larger size, let's say 5 files of a gig each. How to accomplish this?
I'm familiar with the .ebextensions framework, and I think the answer lies by writing an ebextension. The preferred answer changes just the "size" parameter in the existing configuration, without writing in a whole new file, because I'd like to take advantage in any future changes that AWS makes in their docker offering.
If I understand correctly, your aim is to have more history available than the default offers. One way to accomplish this is to enable Log publication to S3. This option is available under Software configuration on the Configuration page for each environment.
This setting will ensure that logs locally rotated out are uploaded to a location in S3. This location is typically under the default bucket that AWS Elastic Beanstalk creates to store artifacts and logs. The bucket name is of the format:
s3://elasticbeanstalk-<region>-<account id>
Ex: s3://elasticbeanstalk-us-east-1-0123456789012
and the path to the logs is:
/resources/environments/logs/publish/<environment id>/<instance id>
Full example:
s3://elasticbeanstalk-us-east-1-0123456789012/resources/environments/logs/publish/e-mpcwnwheky/i-0a1fd158
Things to note
The instance profile that is selected will need the necessary permissions to upload the log files to the S3 bucket at the location specified above.
Logs that are uploaded to S3 are retained there until explicit action is taken such as to delete them or move them to glacier.
Source: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html#health-logs-s3location
I have a postgres dump in AWS S3 bucket, what is the most convenience way to restore it in a AWS RDS ?
AFAIK, there is no native AWS way to manually push data from S3 to anywhere else. The dump stored on S3 needs to first be downloaded and then restored.
You can use the link posted above (http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.Procedural.Importing.html), however that doesn't help you download the data.
The easiest way to get something off of S3 is to simply go to the S3 console and point/click your way to the file, right click it and click Download. If you need to restore FROM an EC2 instance (e.g. because your RDS does not have a public IP), than install and configure the AWS CLI (http://docs.aws.amazon.com/cli/latest/userguide/installing.html).
Once you have the CLI configured, download with the following command:
aws s3 cp s3://<<bucket>>/<<folder>>/<<folder>>/<<key>> dump.gz
NOTE: the above command may need some additional tweaking depending on whether you have multiple AWS profiles installed on the machine, the dump is not one file (but many), etc.
From there restore to RDS just like you would a normal Postgres server following the instructions in the AWS link.
Hope that helps!