I'm migrated from AL1 to AL2 on AWS Beanstalk. AL2 changed location of my nodejs.log to /var/log/{{.}}.stdout.log
I resolved this by adding ryslog.config to .ebexetensions:
files:
"/opt/elasticbeanstalk/config/private/rsyslog.conf.template":
mode: "000644"
owner: root
group: root
content: |
# This rsyslog file redirects Elastic Beanstalk platform logs.
# Logs are initially sent to syslog, but we also want to divide
# stdout and stderr into separate log files.
template(name="SimpleFormat" type="string" string="%msg%\n")
$EscapeControlCharactersOnReceive off
{{range .ProcessNames}}if $programname == '{{.}}' then {
*.=warning;*.=err;*.=crit;*.=alert;*.=emerg /var/log/nodejs/nodejs.log; SimpleFormat
*.=info;*.=notice /var/log/nodejs/nodejs.log; SimpleFormat
}
{{end}}
Above configuration is working but I have problem with log file permissions.
Directory /var/log/nodejs and nodejs.log file are only readable by root (chmod 600), and cloudwatch-agent can't read it. Changing permissions manually do the job, but how can I change the permissions to be created automatically on beanstalk deploy?
Adding the following code resolved it.
This will set the owner and group to the corresponding value, for all files that are automatically created.
# Set the default permissions for all log files
$umask 0022
$FileOwner cwagent
$FileGroup cwagent
$DirOwner cwagent
$DirGroup cwagent
Related
I have an AWS Elasticbeanstalk Python environment and I'd like to change default 1 MB nginx configuration limit for file upload to a bigger value (15 MB).
So I thought to add a config file in .ebextensions:
files:
"/tmp/my.nginx.conf" :
mode: "000755"
owner: root
group: root
content: |
client_max_body_size 15M;
"/tmp/install-nginx-config.sh" :
mode: "000755"
owner: root
group: root
content: |
#!/bin/sh
cp /tmp/my.nginx.conf /etc/nginx/conf.d/elasticbeanstalk/01_max-size.conf
container_commands:
01_runmyshellscript:
command: "sudo /tmp/install-nginx-config.sh"
02_reload_nginx:
command: "sudo service nginx reload"
The idea is to add the new config file in /etc/nginx/conf.d/elasticbeanstalk after the elasticbeanstalk nginx config files are created, so to no to to interfere with the creation of the EB environment, like suggested in many articles on Internet.
But if I deploy the above file in .ebextensions then the environment fails and if I SSH the EC2 I find out that in /etc/nginx/conf.d/elasticbeanstalk there is my 01_max-size.conf but the nginx configuration file created by Elasticbeanstalk isn't there anymore.
This is strange, beacause container_commands should be exetuced after the end of environment creation, so I have no clue of how solve this issue.
I tried the option of adding.platform or .ebextensions folder to the zip file to modify the ngnix setting to allow more than 1MB file but it did not work. I deployed my spring boot as a jar in Elastic Beanstalk. Can you please help me? Appreciate it. Thank you.
Update on the structure. I have used the following directory structures:
1)
root of the zip folder
.platform
ngnix
- conf.d
- ngnix.conf
application.jar
the contents of proxy.conf are:
client_max_body_size 100M;
2)root of the zip folder
.ebextensions
ngnix
- myconf.conf
application.jar
The content of the conf file myconf.conf:
files:
"/etc/nginx/conf.d/proxy.conf":
mode: "000755"
owner: root
group: root
content: |
client_max_body_size 100M;
I have referred to the following links:
https://medium.com/innovation-res/how-to-configure-aws-ebs-to-handle-large-files-10411b9cecf0
https://dev.to/rasoolk16/elastic-beanstalk-update-media-upload-size-53go
https://harishkm.in/2020/08/06/uploading-large-files-to-aws-elastic-beanstalk-fails-with-http-413-request-entity-too-large/
Thank you,
Jeelani
Update:
I've tried keeping the config file directly under .ebextensions and used the following formats to try it out.
files:
"/etc/nginx/conf.d/proxy.conf":
mode: "000755"
owner: root
group: root
content: |
client_max_body_size 100M;
Noticed the following error:
The configuration file __MACOSX/.ebextensions/._01_nginx.config in application version showkase-source-20 contains invalid YAML or JSON. YAML exception: Invalid Yaml: org.yaml.snakeyaml.error.YAMLException: java.nio.charset.MalformedInputException: Input length = 1, JSON exception: Invalid JSON: Unexpected character (�) at position 0.. Update the configuration file.
2) client_max_body_size 100M;
Let me go through the links you have provided and see what am I doing wrong. Thank you. Will keep you posted. But with either of these formats, I don't see the file appearing in /var/app/current. I see only the applicaiton.jar and Procfile.
Contents of config file:
files:
"/etc/nginx/conf.d/01_proxy.conf":
mode: "000644"
owner: root
group: root
content: |
client_max_body_size 20M;
You have used two different methods (via .platform/nginx hook and via .ebextensions). They are both valid, but only one should be needed.
While your platform hook seems fine, the .ebextensions is not according to the AWS documentation. It needs to have the file extension .config (not .conf) and the file needs to lie directly in the .ebextensions/ directory (not .ebextensions/nginx).
Some hints on debugging if this doens't fix your problem:
Check the following CloudWatch logs:
/aws/elasticbeanstalk/<yourAppName>/var/log/eb-engine.log
/aws/elasticbeanstalk/<yourAppName>/var/log/eb-hooks.log
/aws/elasticbeanstalk/<yourAppName>/var/log/nginx/error.log
Log into the EC2 instance via the Session Manager (located at the Systems Manager) to check the following:
Do you see all files of your zip in /var/app/current/?
What is the content of /etc/nginx/conf.d?
If this does not help, you can try the method in this SlackOverflow question.
There, you will run a platform hook script that writes the config file manually.
Edit after you have corrected the above errors:
The contents of your config file seems to be correct. But the error message suggests that you have an invalid file encoding or a special character not matching the encoding (e.g., by copy and pasting from a website). You can use file 01_nginx.config to see the encoding. Best you delete the current config file and write it again by hand (without any copy & pasting) to ensure that no special characters sneak in.
So all I'm trying to do is add "client_max_body_size 5000m;" to my nginx.conf file.
I can ssh in and add the line manually, and that will work, but I know there's a way to do that automatically with my files in the .ebextensions folder.
If I use
files:
"/etc/nginx/nginx.conf":
mode: "000644"
owner: root
group: root
content: |
#Elastic Beanstalk Nginx Configuration File...
nothing seems to change.
if I use
files:
/.platform/nginx/conf.d/proxy.conf:
mode: "000644"
owner: root
group: root
content: |
#Elastic Beanstalk Nginx Configuration File...
I can see that the proxy.conf file is where I would expect it to be, but it still has no impact on my nginx settings, even though the AWS documentation and other posts on here would lead me to believe that EB should copy that file into /etc/nginx/conf.d/elasticbeanstalk/
but it does not, and manually specifying anywhere other than .platform doesn't seem to do anything.
I feel like I'm close, but for the life of me I can't figure it out.
You are trying to use (/etc/nginx/conf.d/proxy.conf) which is for Amazon Linux 1 (AL2). It will not work in AL2 as you should be using .platform/nginx/conf.d/, NOT .ebextentions as shown in the docs.
Therefore, you could have the following .platform/nginx/conf.d/myconfig.conf with content:
client_max_body_size 5000M;
I have a project built in Golang and deployed on a Docker instance in AWS.
Internally I create a log file where the program write several logs.
How can I access that log file?
Is there another correct way to logging?
Thanks
You could mount the log file from your container to your EC2 host. You can do this by using the -v flag when running your container:
docker run -v /var/log/my_host_log_file.log:/var/log/your_container_log_file.log your-image
Alternatively, you can configure your app to log to stdout and use syslog as your log driver (using the --log-driver=syslog switch). Your container logs will then be written to /var/log/messages on your host.
If you use AWS, i would suggest to send Logs direct to AWS CloudWatch.
First create a new Log-Group in AWS Cloudwatch, for example "Production". In your Docker-Compose.yml (or via docker run..) add the AWS Logdriver:
logging:
driver: "awslogs"
options:
awslogs-region: "eu-central-1"
awslogs-group: "Production"
awslogs-stream: "MyApp"
Next creat a IAM user with Access to AWS Cloudwatch and add to the Dockerhost the credentials.
Example IAM Policy:
"Version" "2012-10-17"
"Statement"
"Action" "logs:CreateLogStream" "logs:PutLogEvents" "Effect" "Allow" "Resource"
On Ubuntu with systemd:
"Version" "2012-10-17"
"Statement"
"Action"
"logs:CreateLogStream"
"logs:PutLogEvents"
"Effect"
"Allow" "Resource"
And add to the File:
[Service] Environment"AWS_ACCESS_KEY_ID=<aws_access_key_id>"
Environment"AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>"
Run:
systemctl daemon-reload
service docker restart
Now your logs should appear in AWS Cloudwatch.
Thanks for reply.
After a while looking for the solution to the problem, I found it!
Firstly, I needed to mount the file that is inside the instance in the docker-host.
To do this I add a Json file in the root folder of my project called Dockerrun.aws.json
( http://docs.aws.amazon.com/es_es/elasticbeanstalk/latest/dg/create_deploy_docker_image.html#create_deploy_docker_image_dockerrun )
That is the file that declares the shared folder (volumes) (beetwen docker-host and instance) where I save my log file . This line is equivalent to adding -v flag in the docker run command (https://docs.docker.com/engine/tutorials/dockervolumes/#mount-a-host-directory-as-data-volume). I do this this way because I can not add mount to a running instance and i cant stop it by ssh.
{
"AWSEBDockerrunVersion": "1",
"Volumes": [
{
"HostDirectory": "/var/log/",
"ContainerDirectory": "/go/src/app/log"
}
]
}
Then to tell aws that I want to download my log file when I request records. (Tail (last 100 lines), bundle or rotate) I add these files to the .ebextension folder in my project directory. ( http://docs.aws.amazon.com/en_us/elasticbeanstalk/latest/dg/using-features.logging.html#health-logs-extend )
Log_bundle.conf
Files:
"/opt/elasticbeanstalk/tasks/bundlelogs.d/log_bundle.conf":
Mode: "000755"
Owner: root
Group: root
Content: |
/var/log/application.log
Log_rotate.config
Files:
"/opt/elasticbeanstalk/tasks/bundlelogs.d/log_rotate.conf":
Mode: "000755"
Owner: root
Group: root
Content: |
/var/log/application.log
Log_tail.config
Files:
"/opt/elasticbeanstalk/tasks/publishlogs.d/log_tail.conf":
Mode: "000755"
Owner: root
Group: root
Content: |
/var/log/application.log
Finally, I dont try Amazon Could Watch but is the next step.
Regards
If you use ELK (Elasticsearch, Logstash, Kibana), I would suggest to use "logrus"
Get the library
go get github.com/sirupsen/logrus
Then in your project
package main
import (
logrus "github.com/sirupsen/logrus"
)
var log = logrus.New()
func main() {
conn, _ := net.Dial("tcp", "logstash-address")
hook := logrustash.New(conn, logrustash.DefaultFormatter(logrus.Fields{"type": "my-app"}))
log.Hooks.Add(hook)
log.Info("Hello World!")
}
It seems like by default Django logs errors to apache's error file. On Ubuntu and apache2 this file is /var/log/apache2/error.log and the permissions on the /var/log/apache2 directory is 750 with root as the owner and adm as the group. The permissions on the error.log file itself is 640 with root as the owner and adm as the group.
When errors occur, you can see the stack trace in the error.log file. I want to have django errors in a custom log file, not apache's error log, so I setup the logging config in settings.py. I have a /var/log/example directory and a /var/log/example/myapperrors.log file. I set the permissions exactly like apache2 log directory and error file. However, I get permission denied on the myapperrors.log file.