I'm new to AWS CLI (and programming), but I've looked through documentation and posted questions and can't find this addressed, I must be missing something basic?
How do I save the output? I'd like to run AWS S3 Sync to backup my data overnight, and I'd like to see a log report in the morning of what happened.
At this point, I can run AWS from a command prompt:
aws s3 sync "my local directory" s3://mybucket
I've set output format to Text in the config. But I'm only seeing the text in the command prompt. How can I export it as a log file?
Is this not possible, what am I missing?
Many thanks in advance,
Matthew
aws s3 sync "my local directory" s3://mybucket --debug 2> "local path\logname.txt"
Not only did I figure out adding > filename to the end of the command, but I also figured out that when saving this as a batch file, it won't run as a scheduled task in Windows Server 2008 r2, or Windows 7, if it contains drive mappings. UNC paths are required.
Thanks!
Matthew
this perfectly worked for me
aws cloudformation describe-stack-events --stack-name "stack name" --debug 2> "C:\Users\ravi\Desktop\CICDWORKFolder\RedshiftFolder\logname.txt"
Related
I am using the aws s3 cp command to upload files to my s3 bucket. I am using windows cmd for this and when I run the following command:
aws s3 cp "logs.txt" s3://test/logs.txt --debug > ./log_test.txt
With the debug flag I see a lot of relevant information which is showed in the cmd window but when I open the log_test.txt file I can only see one line (maybe not even one).
Is there a way to save the full output showed in my cmd screen into a file?
This issue also happens when an error occurs. So if my file is not uploaded correctly I can't debug what the error was. Which is the main pain point I have.
I have already tried adding the --output flag but this only allows you to change the format of the output which is still not saved.
If you want to store complete log, the one that you see when your run that command without redirection then you can use &> instead of > for unix based systems.
So it should look like
aws s3 cp "logs.txt" s3://test/logs.txt --debug &> ./log_test.txt
I am not 100% sure what is the Windows equivalent, but I think it is 2> so you can try
aws s3 cp "logs.txt" s3://test/logs.txt --debug 2> ./log_test.txt
I'm having a problem using AWS Device Farm, but the problem is that Amazon is not very specific on what goes wrong.
After I created a new run and try to upload my apk file it shows this message before getting to finish the upload:
There was a problem uploading your file. Please try again.
There are no error codes. I have already tried several times using a signed app for debug and for release, but neither of them finishes the upload. Is this a temporal problem in Amazon cloud or it is a known error?
I work for the AWS Device Farm team.
Sorry to hear that you are running in to issues.
If it is the App that is giving you an error you should check if you are able to run the app locally on a real device. If yes, then this should be working on device farm. At times, app build for emulators/simulators are uploaded and can cause the error.
If it is the test apk that you are uploading then the same thing as point 1 should be confirmed.
If both of the points above are true and you are still getting an error please start a thread on AWS Device Farm forums and we can take a closer look at your runs or you can share your run url here and we can take a look.
Would it be possible to try and upload this file using the CLI[1]? The create-upload command will do the same thing the web console is doing and it can return more information than the web console.
aws devicefarm create-upload --project-arn <yourProjectsArn> --name <nameOfFile> --type <typeOfAppItIs> --region us-west-2
This will return a upload-arn which you will need to use later so keep it handy. If you need a more verbosity on any of the CLI commands listed here you can use the --debug option.
The create-upload command will return a presigned-url which you can do a PUT command on.
curl:
curl -T someAppFileWithSameNameAsSpecifiedBefore "presigned-url"
Once you have the file now uploaded you can do a get-upload command to see the status of the upload and if there are any problems this will show why.
aws devicefarm get-upload --arn <uploadArnReturnToYouFromPreviousCommand> --region us-west-2
My output looks like this:
{
"upload": {
"status": "SUCCEEDED",
"name": "app-debug.apk",
"created": 1500080938.105,
"type": "ANDROID_APP",
"arn": "arn:aws:devicefarm:us-west-2:<accountNum>:upload:<uploadArn>",
"metadata": "{\"device_admin\":false,\"activity_name\":\"com.xamarin.simplecreditcardvalidator.MainActivity\",\"version_name\":\"1.1\",\"screens\":[\"small\",\"normal\",\"large\",\"xlarge\"],\"error_type\":null,\"sdk_version\":\"21\",\"package_name\":\"com.xamarin.simplecreditcardvalidator\",\"version_code\":\"2\",\"native_code\":[],\"target_sdk_version\":\"25\"}"
}}
Please let me know what this returns and I look forward to your response.
Best Regards
James
[1] http://docs.aws.amazon.com/cli/latest/reference/devicefarm/create-upload.html
Also used this article to learn how to do most of this:
https://aws.amazon.com/blogs/mobile/get-started-with-the-aws-device-farm-cli-and-calabash-part-1-creating-a-device-farm-run-for-android-calabash-test-scripts/
I have a cluster up and running. I am trying to add a step to run my code. The code itself works fine on a single instance. Only thing is I can't get it to work off S3.
aws emr add-steps --cluster-id j-XXXXX --steps Type=spark,Name=SomeSparkApp,Args=[--deploy-mode,cluster,--executor-memory,0.5g,s3://<mybucketname>/mypythonfile.py]
This is exactly what examples show I should do. What am I doing wrong?
Error I get:
Exception in thread "main" java.lang.IllegalArgumentException: Unknown/unsupported param List(--executor-memory, 0.5g, --executor-cores, 2, --primary-py-file, s3://<mybucketname>/mypythonfile.py, --class, org.apache.spark.deploy.PythonRunner)
Usage: org.apache.spark.deploy.yarn.Client [options]
Options:
--jar JAR_PATH Path to your application's JAR file (required in yarn-cluster
mode)
.
.
.
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Command exiting with ret '1'
When I specify as this instead:
aws emr add-steps --cluster-id j-XXXXX --steps Type=spark,Name= SomeSparkApp,Args=[--executor-memory,0.5g,s3://<mybucketname>/mypythonfile.py]
I get this error instead:
Error: Only local python files are supported: Parsed arguments:
master yarn-client
deployMode client
executorMemory 0.5g
executorCores 2
EDIT: IT gets further along when I manually create the python file after SSH'ing into the cluster, and specifying as follows:
aws emr add-steps --cluster-id 'j-XXXXX' --steps Type=spark,Name= SomeSparkApp,Args=[--executor-memory,1g,/home/hadoop/mypythonfile.py]
But, not doing the job.
Any help appreciated. This is really frustrating as a well documented method on AWS's own blog here https://blogs.aws.amazon.com/bigdata/post/Tx578UTQUV7LRP/Submitting-User-Applications-with-spark-submit does not work.
I will ask, just in case, you used your correct buckets and cluster ID-s?
But anyways, I had similar problems, like I could not use --deploy-mode,cluster when reading from S3.
When I used --deploy-mode,client,--master,local[4] in the arguments, then I think it worked. But I think I still needed something different, can't remember exactly, but I resorted to a solution like this:
Firstly, I use a bootstrap action where a shell script runs the command:
aws s3 cp s3://<mybucket>/wordcount.py wordcount.py
and then I add a step to the cluster creation through the SDK in my Go application, but I can recollect this info and give you the CLI command like this:
aws emr add-steps --cluster-id j-XXXXX --steps Type=CUSTOM_JAR,Name="Spark Program",Jar="command-runner.jar",ActionOnFailure=CONTINUE,Args=["spark-submit",--master,local[4],/home/hadoop/wordcount.py,s3://<mybucket>/<inputfile.txt> s3://<mybucket>/<outputFolder>/]
I searched for days and finally discovered this thread which states
PySpark currently only supports local
files. This does not mean it only runs in local mode, however; you can
still run PySpark on any cluster manager (though only in client mode). All
this means is that your python files must be on your local file system.
Until this is supported, the straightforward workaround then is to just
copy the files to your local machine.
Is there any way to disable the
Completed 1 of 12 part(s) with 11 file(s) remaining...
progress output with the aws s3 sync command (from the aws cli tools).
I know there is a --quiet option but I don't want to use it because I still want the Upload... details in my logfile.
Not a big issue, but creates mess in the logfile like:
Completed 1 of 12 part(s) with 11 file(s) remaining^Mupload: local/file to s3://some.bucket/remote/file
Where ^M is a control character.
As of October 2017, it is possible to only suppress upload progress with aws s3 cp and aws s3 sync by using the --no-progress option:
--no-progress (boolean) File transfer progress is not displayed. This flag is only applied when the quiet and only-show-errors flags are not
provided.
Example:
aws s3 sync /path/to/directory s3://bucket/folder --no-progress
Output:
upload: /path/to/directory to s3://bucket/folder
I had a quick look at the CLI tools code and currently it is not possible to disable that message.
You should use --only-show-errors flag while running the command. Also, you would want --no-progress. This is going to minimize the logging.
More specs: https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
You can't disable the message completely. You can only delete by editing but when you run again, it would show up again.
I would like to ask about a processing task I am trying to complete using a data pipeline in AWS, but I have not been able to get it to work.
Basically, I have 2 data nodes representing 2 MySQL databases, where the data is supposed to be extracted from periodically and placed in an S3 bucket. This copy activity is working fine selecting daily every row that has been added, let's say today - 1 day.
However, that bucket containing the collected data as CSVs should become the input for an EMR activity, which will be processing those files and aggregating the information. The problem is that I do not know how to remove or move the already processed files to a different bucket so I do not have to process all the files every day.
To clarify, I am looking for a way to move or remove already processed files in an S3 bucket from a pipeline. Can I do that? Is there any other way I can only process some files in an EMR activity based on a naming convention or something else?
Even better, create a DataPipeline ShellCommandActivity and use the aws command line tools.
Create a script with these two lines:
sudo yum -y upgrade aws-cli
aws s3 rm $1 --recursive
The first line ensures you have the latest aws tools.
The second one removes a directory and all its contents. The $1 is an argument passed to the script.
In your ShellCommandActivity:
"scriptUri": "s3://myBucket/scripts/theScriptAbove.sh",
"scriptArgument": "s3://myBucket/myDirectoryToBeDeleted"
The details on how the aws s3 command works are at:
http://docs.aws.amazon.com/cli/latest/reference/s3/index.html
1) Create a script which takes input path and then deletes the files using hadoop fs -rmr s3path.
2) Upload the script to s3
In emr use the prestep -
1) hadoop fs -copyToLocal s3://scriptname .
2) chmod +x scriptname
3) run script
That pretty much it.
Another approach without using EMR is to install s3cmd tool through ShellCommandActivity in a small EC2 instance, then you can use s3cmd in pipeline to operate your S3 repo in whatever way you want.
A tricky part of this approach is to configure s3cmd through a configuration file safely (basically pass access key and secret), as you can't just ssh into the EC2 instance and use 's3cmd --configure' interactively in a pipeline.
To do that, you create a config file in the ShellCommandActivity using 'cat'. For example:
cat <<EOT >> s3.cfg
blah
blah
blah
EOT
Then use '-c' option to attach the config file every time you call s3cmd like this:
s3cmd -c s3.cfg ls
Sounds complicated, but works.