Is there any way to disable the
Completed 1 of 12 part(s) with 11 file(s) remaining...
progress output with the aws s3 sync command (from the aws cli tools).
I know there is a --quiet option but I don't want to use it because I still want the Upload... details in my logfile.
Not a big issue, but creates mess in the logfile like:
Completed 1 of 12 part(s) with 11 file(s) remaining^Mupload: local/file to s3://some.bucket/remote/file
Where ^M is a control character.
As of October 2017, it is possible to only suppress upload progress with aws s3 cp and aws s3 sync by using the --no-progress option:
--no-progress (boolean) File transfer progress is not displayed. This flag is only applied when the quiet and only-show-errors flags are not
provided.
Example:
aws s3 sync /path/to/directory s3://bucket/folder --no-progress
Output:
upload: /path/to/directory to s3://bucket/folder
I had a quick look at the CLI tools code and currently it is not possible to disable that message.
You should use --only-show-errors flag while running the command. Also, you would want --no-progress. This is going to minimize the logging.
More specs: https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
You can't disable the message completely. You can only delete by editing but when you run again, it would show up again.
Related
How does the AWS CLI command completion setting cli_auto_prompt = on-partial work? In my ~/.aws/config I have
[default]
region = us-west-2
output = json
cli_auto_prompt = on-partial
cli_history = enabled
cli_timestamp_format = iso8601
But when I enter complete commands such as
aws iam list-groups
and press enter, I still get the completion UI. As I understand the documentation, which says
on-partial uses partial auto-prompt mode. If a command is incomplete or cannot be run due to client-side validation errors, auto-prompt is used. This mode is particular useful if you have pre-existing scripts, runbooks, or you only want to be auto-prompted for commands you are unfamiliar with rather than prompted on every command.
I should not have had the completion UI interrupt me, and instead the command should just have been executed.
How do I avoid the AWS CLI completion UI for complete commands?
(The other settings, such as output, and cli_history all work as expected, and respond to changes made in ~/.aws/config.)
$ aws --version
aws-cli/2.7.11 Python/3.10.5 Darwin/20.6.0 source/x86_64 prompt/on
$ bash --version
GNU bash, version 5.1.16(1)-release (x86_64-apple-darwin20.6.0)
$ which -a aws
/usr/local/bin/aws
$ ls -l /usr/local/bin/aws
lrwxr-xr-x 1 Tax admin 31 Jun 25 09:50 /usr/local/bin/aws -> ../Cellar/awscli/2.7.11/bin/aws
Check whether you have the AWS_CLI_AUTO_PROMPT environment variable set somewhere in your ~/.bashrc, ~/.bash_profile, or similar files.
As you can see in the AWS CLI auto-prompt documentation, this variable overrides the cli_auto_prompt setting in the ~/.aws/config file.
If things still look spooky after that, try completely uninstalling and reinstalling the AWS CLI.
Linux Distro: Pop!_OS 19.10 Ubuntu
For context, I'm hosting a static website through Google Cloud Bucket.
So I tried executing the command
gsutil -m cp -r ejsout gs://evaapp.xyz
to push to my storage bucket.
The command copies all the files successfully, returning
/ [333/333 files][ 19.7 MiB/ 19.7 MiB] 100% Done 998.4 KiB/s ETA 00:00:00
Operation completed over 333 objects/19.7 MiB.
but when I go and look at the bucket online, the files aren't overwritten.
I've waited for hours or days and nothing happens to the files on Google Cloud Console. Only new files that haven't been created show up in the bucket console but not existing files. I have to manually update the files through cloud console for it to update.
Anyway to fix this? Feel free to close this issue because I'm bad at this and I couldn't find any helpful documentation on this. It's just annoying, I want to do the push from the terminal. Thanks!
I am using the aws s3 cp command to upload files to my s3 bucket. I am using windows cmd for this and when I run the following command:
aws s3 cp "logs.txt" s3://test/logs.txt --debug > ./log_test.txt
With the debug flag I see a lot of relevant information which is showed in the cmd window but when I open the log_test.txt file I can only see one line (maybe not even one).
Is there a way to save the full output showed in my cmd screen into a file?
This issue also happens when an error occurs. So if my file is not uploaded correctly I can't debug what the error was. Which is the main pain point I have.
I have already tried adding the --output flag but this only allows you to change the format of the output which is still not saved.
If you want to store complete log, the one that you see when your run that command without redirection then you can use &> instead of > for unix based systems.
So it should look like
aws s3 cp "logs.txt" s3://test/logs.txt --debug &> ./log_test.txt
I am not 100% sure what is the Windows equivalent, but I think it is 2> so you can try
aws s3 cp "logs.txt" s3://test/logs.txt --debug 2> ./log_test.txt
I'm new to AWS CLI (and programming), but I've looked through documentation and posted questions and can't find this addressed, I must be missing something basic?
How do I save the output? I'd like to run AWS S3 Sync to backup my data overnight, and I'd like to see a log report in the morning of what happened.
At this point, I can run AWS from a command prompt:
aws s3 sync "my local directory" s3://mybucket
I've set output format to Text in the config. But I'm only seeing the text in the command prompt. How can I export it as a log file?
Is this not possible, what am I missing?
Many thanks in advance,
Matthew
aws s3 sync "my local directory" s3://mybucket --debug 2> "local path\logname.txt"
Not only did I figure out adding > filename to the end of the command, but I also figured out that when saving this as a batch file, it won't run as a scheduled task in Windows Server 2008 r2, or Windows 7, if it contains drive mappings. UNC paths are required.
Thanks!
Matthew
this perfectly worked for me
aws cloudformation describe-stack-events --stack-name "stack name" --debug 2> "C:\Users\ravi\Desktop\CICDWORKFolder\RedshiftFolder\logname.txt"
I would like to ask about a processing task I am trying to complete using a data pipeline in AWS, but I have not been able to get it to work.
Basically, I have 2 data nodes representing 2 MySQL databases, where the data is supposed to be extracted from periodically and placed in an S3 bucket. This copy activity is working fine selecting daily every row that has been added, let's say today - 1 day.
However, that bucket containing the collected data as CSVs should become the input for an EMR activity, which will be processing those files and aggregating the information. The problem is that I do not know how to remove or move the already processed files to a different bucket so I do not have to process all the files every day.
To clarify, I am looking for a way to move or remove already processed files in an S3 bucket from a pipeline. Can I do that? Is there any other way I can only process some files in an EMR activity based on a naming convention or something else?
Even better, create a DataPipeline ShellCommandActivity and use the aws command line tools.
Create a script with these two lines:
sudo yum -y upgrade aws-cli
aws s3 rm $1 --recursive
The first line ensures you have the latest aws tools.
The second one removes a directory and all its contents. The $1 is an argument passed to the script.
In your ShellCommandActivity:
"scriptUri": "s3://myBucket/scripts/theScriptAbove.sh",
"scriptArgument": "s3://myBucket/myDirectoryToBeDeleted"
The details on how the aws s3 command works are at:
http://docs.aws.amazon.com/cli/latest/reference/s3/index.html
1) Create a script which takes input path and then deletes the files using hadoop fs -rmr s3path.
2) Upload the script to s3
In emr use the prestep -
1) hadoop fs -copyToLocal s3://scriptname .
2) chmod +x scriptname
3) run script
That pretty much it.
Another approach without using EMR is to install s3cmd tool through ShellCommandActivity in a small EC2 instance, then you can use s3cmd in pipeline to operate your S3 repo in whatever way you want.
A tricky part of this approach is to configure s3cmd through a configuration file safely (basically pass access key and secret), as you can't just ssh into the EC2 instance and use 's3cmd --configure' interactively in a pipeline.
To do that, you create a config file in the ShellCommandActivity using 'cat'. For example:
cat <<EOT >> s3.cfg
blah
blah
blah
EOT
Then use '-c' option to attach the config file every time you call s3cmd like this:
s3cmd -c s3.cfg ls
Sounds complicated, but works.