I am trying to host a shiny app on an AWS EC2 for the first time. I have been following this [tutorial] (https://www.charlesbordet.com/en/guide-shiny-aws/#3-how-to-configure-shiny-server).
I adjusted my sudo nano /etc/shiny-server/shiny-server.conf with sanitize_errors false; so the errors display at http://18.144.34.215:3838/. It seems I do not have the correct permission allocated to that folder from the shiny-server.
This is my first attempt at hosting a shiny app on EC2 and a bit lost from other posts I have found searching. What would be the correct commands to give permission to this folder?
Also, please let me know what info you need from me in order to understand this error better.
Here are my folder permission for 'RIBBiTR_DataRepository'
-rw-rwSr-- 1 ubuntu ubuntu 35149 Feb 1 21:32 LICENSE
-rw-rwSr-- 1 ubuntu ubuntu 10 Feb 1 21:32 README.md
drwxrwsrwx 5 ubuntu ubuntu 4096 Feb 1 21:38 RIBBiTR_DataRepository
-rw-rwSr-- 1 ubuntu ubuntu 205 Feb 1 21:32 db_forms.Rproj
drwxrwsr-x 2 ubuntu ubuntu 4096 Feb 1 21:32 misc
And to add, when I try to view the logs I receive a permission denied
ubuntu#ip-172-30-1-21:/var/log/shiny-server$ sudo tail RIBBiTR_DataRepository-shiny-20230201-215702-44689.log
su: ignoring --preserve-environment, it's mutually exclusive with --login
-bash: line 1: cd: /srv/shiny-server/db_forms/RIBBiTR_DataRepository: Permission denied
The issue was in my shiny-server.conf file. I updated the file with a user group, run_as ubuntu;
Files copied by SageMaker from s3 to docker inside ProcessingJob have root ownership and permissions which do not allow non owner users (non root) to write to them.
I'd like to run the docker container as non root user and be able to write to folders created by SageMaker, so the Dockerfile looks like this:
FROM base
...
USER nonroot
Exemplary permissions and ownership of folders copied from s3 to SageMaker's docker container:
2022-11-30T10:20:13.567+01:00 + ls -la /opt/ml/processing
2022-11-30T10:20:13.567+01:00 total 24
2022-11-30T10:20:13.567+01:00 drwxr-xr-x 6 root root 4096 Nov 30 09:20 .
2022-11-30T10:20:13.567+01:00 drwxr-xr-x 5 root root 4096 Nov 30 09:20 ..
2022-11-30T10:20:13.567+01:00 drwxr-xr-x 2 root root 4096 Nov 30 09:20 data
2022-11-30T10:20:13.568+01:00 drwxr-xr-x 2 root root 4096 Nov 30 09:20 output
I'd expect these folders to either has nonroot user ownership.
I've checked the documentation but no luck there. If there's any obvious way to achieve this that I missed, please let me know. Thanks!
Using steps provided here, I kicked off a CodeBuild with the following advanced options checked:
Enable session connection
Allow AWS CodeBuild to modify this service role so it can be used with this build project
The buildspec included a codebuild-breakpoint:
version: 0.2
phases:
pre_build:
commands:
- ls -al
- codebuild-breakpoint
- cd "${SERVICE_NAME}"
- ls -al
- $(aws ecr get-login)
- TAG="$SERVICE_NAME"
build:
commands:
- docker build --tag "${REPOSITORY_URI}:${TAG}" .
post_build:
commands:
- docker push "${REPOSITORY_URI}:${TAG}"
- printf '{"tag":"%s"}' $TAG > ../build.json
artifacts:
files: build.json
The build started and produced the following logs without pausing:
[Container] 2022/02/28 13:49:03 Entering phase PRE_BUILD
[Container] 2022/02/28 13:49:03 Running command ls -al
total 148
drwxr-xr-x 2 root root 4096 Feb 28 13:49 .
drwxr-xr-x 3 root root 4096 Feb 28 13:49 ..
-rw-rw-rw- 1 root root 1818 Feb 28 10:54 user-manager\Dockerfile
-rw-rw-rw- 1 root root 140 Feb 28 10:34 user-manager\body.json
-rw-rw-rw- 1 root root 0 Feb 28 10:54 user-manager\shared-modules\
-rw-rw-rw- 1 root root 4822 Feb 21 14:52 user-manager\shared-modules\config-helper\config.js
-rw-rw-rw- 1 root root 2125 Feb 21 14:52 user-manager\shared-modules\config-helper\config\default.json
-rw-rw-rw- 1 root root 366 Feb 21 14:52 user-manager\shared-modules\config-helper\package.json
-rw-rw-rw- 1 root root 9713 Feb 21 14:52 user-manager\shared-modules\dynamodb-helper\dynamodb-helper.js
-rw-rw-rw- 1 root root 399 Feb 21 14:52 user-manager\shared-modules\dynamodb-helper\package.json
-rw-rw-rw- 1 root root 451 Feb 21 14:52 user-manager\shared-modules\token-manager\package.json
-rw-rw-rw- 1 root root 13885 Feb 21 14:52 user-manager\shared-modules\token-manager\token-manager.js
-rw-rw-rw- 1 root root 44372 Feb 28 10:34 user-manager\src\cognito-user.js
-rw-rw-rw- 1 root root 706 Feb 28 10:34 user-manager\src\package.json
-rw-rw-rw- 1 root root 32734 Feb 28 10:34 user-manager\src\server.js
[Container] 2022/02/28 13:49:03 Running command codebuild-breakpoint
2022/02/28 13:49:03 Build is paused temporarily and you can use codebuild-resume command in the session to resume this build
[Container] 2022/02/28 13:49:03 Running command cd "${SERVICE_NAME}"
/codebuild/output/tmp/script.sh: 4: cd: can't cd to user-manager
My primary question is: Why didn't the build pause and session manager link become available?
Side-quest: The reason I'm trying to debug the session is to try to determine why the process can't CD to the user-manager folder (which clearly exists). Any ideas why?
TLDR: The image on the build machine was too old.
Main quest
The template specified aws/codebuild/ubuntu-base:14.04 as the CodeBuild image. Presumably that image pre-dated the Session Manager functionality (which requires a specific version of the SSM agent to be installed).
I update the agent to aws/codebuild/standard:5.0 and was able to successfully pause on the breakpoint and connect to the session.
Side quest
Once I connected I was able to investigate the cause of the inability to CD to the folder. I can confirm that Tim's shot in the dark was correct! All the entries were in fact files - no folders.
This QuickStart is the gift that keeps on giving! When/if I get all the issues resolved I'll submit a PR to update the project. Those interested in the cause of the file/folder issue can follow up there.
Side quest update
The strange flattening behaviour was due to creating the zip file on a Windows machine and unzipping it on a unix machine (the build agent uses an Ubuntu image). Just zipped it using 7-Zip and that did the job.
I am trying to logrotate my log files. Here is my configuration file:
/home/deploy/apps/production_app/current/log/*.log {
daily
missingok
rotate 52
compress
create 0644 deploy deploy
delaycompress
notifempty
sharedscripts
copytruncate
}
And this is result of
ll apps/production_app/current/log/
on my log files:
-rw-rw-r-- 1 deploy deploy 0 Jul 1 10:01 production.log
-rw-rw-r-- 1 deploy deploy 1124555 Jul 1 10:01 production.log.1
And when I run this command
logrotate -v /etc/logrotate.d/production_app
I get following:
error: error creating output file /var/lib/logrotate.status.tmp:
Permission denied
And here is permission on my log-rotate config file
lrwxrwxrwx 1 root root 67 Feb 25 2019 /etc/logrotate.d/production_app -> /home/deploy/apps/production_app/shared/config/log_rotation
please check whether the dir "var/lib" is readonly.
I am attempting to run a few scripts while deploying using AWS Code Deploy, but they never run due to not having permissions to run the scripts.
Here is my appspec.yml file:
version: 0.0
os: linux
files:
- source: /
destination: /var/www/html
permissions:
- object: /var/www/html/codedeploy-scripts
owner: root
mode: 777
type:
- directory
hooks:
ApplicationStop:
- location: codedeploy-scripts/application-stop
timeout: 300
runas: root
BeforeInstall:
- location: codedeploy-scripts/before-install
timeout: 300
runas: root
AfterInstall:
- location: codedeploy-scripts/after-install
timeout: 600
runas: root
ApplicationStart:
- location: codedeploy-scripts/application-start
timeout: 300
runas: root
ValidateService:
- location: codedeploy-scripts/validate-service
timeout: 300
runas: root
The codedeploy-scripts folder get deployed with the app and the permissions I set on the folder does not get set. The permissions on the folder always get reset to:
[ec2-user#ip-10-0-8-181 html]$ ls -al
total 156
drwxrwsr-x 7 ec2-user www 4096 Oct 13 16:36 .
drwxrwsr-x 3 ec2-user www 4096 Oct 13 15:01 ..
-rw-rw-r-- 1 ec2-user www 740 Oct 13 16:28 appspec.yml
drwxr-sr-x 2 ec2-user www 4096 Oct 13 16:36 codedeploy-scripts
...
The files in the folder seem to have executable rights:
[ec2-user#ip-10-0-8-181 alio]$ ls -al codedeploy-scripts
total 28
drwxr-sr-x 2 ec2-user www 4096 Oct 13 16:36 .
drwxrwsr-x 7 ec2-user www 4096 Oct 13 16:36 ..
-rwxr-xr-x 1 ec2-user www 343 Oct 13 16:28 after-install
-rwxr-xr-x 1 ec2-user www 12 Oct 13 16:28 application-start
-rwxr-xr-x 1 ec2-user www 12 Oct 13 16:28 application-stop
-rwxr-xr-x 1 ec2-user www 889 Oct 13 16:28 before-install
-rwxr-xr-x 1 ec2-user www 12 Oct 13 16:28 validate-service
Why doesn't the code get deployed with the permissions i set in the appspec file. The codedeploy-scripts folder should have 777 permissions but it never does.
This is the error i get in /var/log/aws/codedeploy-agent/codedeploy-agent.log for each of those scripts:
2015-10-13 16:36:23 WARN [codedeploy-agent(9918)]: InstanceAgent::Plugins::CodeDeployPlugin::HookExecutor: Script at specified location: codedeploy-scripts/validate-service is not executable. Trying to make it executable.
Any help would be appreciated.
The agent is executing the scripts directly from the extracted archive bundle not from any arbitrary places you might have copied them using the files section. You'll need to set the execute bit in your archive in S3 or Git repository.
What you have as is does this:
Copy all the files to /var/www/html.
Set permissions on the directory on the contents of /var/www/html/codedeploy-scripts to 777 but not the directory itself (See the appspec.yml reference). This will also be affected by umask, which you might be setting /etc/profile.
Execute each of the scripts for the lifecycle events (as they occur) from the archive root. So your ValidateSerivce script is running from <deployment-archive-root>/codedeploy-scripts/validate-service not from /var/www/html/codedeploy-scripts/validate-service
Note: ApplicationStop is special because it runs before new new archive bundle is downloaded.
Without more details, I won't be able to speak to why setting your scripts to be executable fixed your issue, but the accepted answer shouldn't have resolved anything other than the log statement you were seeing.
Take a closer look at the log:
2015-10-13 16:36:23 WARN [codedeploy-agent(9918)]: InstanceAgent::Plugins::CodeDeployPlugin::HookExecutor: Script at specified location: codedeploy-scripts/validate-service is not executable. Trying to make it executable.
It's only a warning, not an error. The Code Deploy agent noticed that your validate_service.sh script wasn't executable and it was "Trying to make it executable". If we look at the relevant Code Deploy agent code, you'll see that the agent will chmod +x the script itself.
When you set your scripts to be executable, you only silenced this warning, and it shouldn't have affected anything else. Looking back at the Code Deploy agent code, in L106, if the agent wasn't able to make your scripts executable you would have seen an error in your logs.
To answer your question on the permissions, you have a misconfigured appspec.yml. When you say:
permissions:
- object: /var/www/html/codedeploy-scripts
owner: root
mode: 777
type:
- directory
You are telling Code Deploy to set all files of type "directory" within /var/www/html/codedeploy-scripts to have permissions 777.
All of your scripts under codedeploy-scripts are "file" types (not "directory"), which is why their permissions weren't set, and the permissions only apply to files under the directory you specify, which is why the permissions on the codedeploy-scripts directory weren't set.
Here's the description of the appspec.yml permission's type option from the AWS docs:
type – Optional. The types of objects to apply the specified permissions to. This can be set to file or directory. If file is specified, the permissions will be applied only to files that are immediately contained within object after the copy operation (and not to object itself). If directory is specified, the permissions will be recursively applied to all directories/folders that are anywhere within object after the copy operation (but not to object itself).
I'd like to expand on an issue mentioned by Jonathan Turpie which can create a very weird situation.
From the docs on ApplicationStop:
This deployment lifecycle event occurs even before the application revision is downloaded. ... The AppSpec file and scripts used for this deployment lifecycle event are from the previous successfully deployed application revision.
Now imagine this situation:
A revision was deployed with botched ApplicationStop script permissions. The deployment still went fine because a previous version was used.
A new revision is pushed and fails the ApplicationStop step (because now it tried to execute the botched script from step 1).
You notice your mistake, fix the code, publish a new revision, but it still fails with the same error!
At this point it's not possible to fix the error by deploying new code. You only have two options:
In the deployment settings enable "Ignore Stop failures" (e.g. with the --ignore-application-stop-failures CLI flag [1])
Manually fix the file permissions in the previous successful deployment's root.
This concerns any stop script failures, not just permissions of course.
[1] https://docs.aws.amazon.com/cli/latest/reference/deploy/create-deployment.html
Solving permission issues:
Hoping you are in the root directory where all your scripts .sh files reside:
chmod +x ./*.sh
This makes all .sh files executable
Add some script change_permissions.sh and add the following in the file:
#!/bin/bash
chmod -R 777 /var/www/html/
This will give your destination folder - /var/www/html/ executable permissions.
Finally asspec.yml file add somehow following:
BeforeInstall:
- location: change_permissions.sh
timeout: 6
runas: root
This will at run time in your ec2 instance apply the executable permission to the files.