I have in Airflow a BashOperator that executes an aws s3 ls s3:/path command. It works on the console but it doesn't works on Airflow, the error message is: command not found . Aws is correctly installed, the following URL explains how the PATH was set:
How do I resolve the "-bash: aws: command not found" awscli error? and how it was installed: Is it possible to install aws-cli package without root permission? .
aws --version
aws-cli/1.18.209 Python/3.8.5 Linux/5.4.0-58-generic botocore/1.19.49
I dont know what I am doing wrong. Please help. How can I make this command (and any other in aws) work on Airflow ?
Thanks in advance.
Related
I just installed aws cli on Ubuntu following the oficial installation guide on an azure VM.
When I run any command from the command line the results are python objects and not a text or regular output
$ aws s3 ls
<botocore.awsrequest.AWSRequest object at 0x7f412f3573a0>
I searched everywhere but I cant find any hint.
I already reinstalled aws and also tried using the output flag but nothing changes.
Any suggestions?
This took me a while to figure out as well. For some reason this only affected our CICD jobs, but using the same exact container image and env vars locally worked fine.
Turns out, the issue stems from not providing a region.
You can fix this by specifying the region explicitly in the command:
aws s3 ls --region us-west-2
Or by providing the region with the available AWS env vars:
export AWS_REGION="us-west-2"
# or
export AWS_DEFAULT_REGION="us-west-2"
Some related sources that helped me figure this out:
https://github.com/jwalton/gh-ecr-login/issues/3
aws s3 ls gives error botocore.utils.BadIMDSRequestError: <botocore.awsrequest.AWSRequest object at 0x7f3f6cb44d00>
Well, I don't know how I didn't tried this before, but installing the awscli with apt fixed the issue.
sudo apt-get install awscli.
Thank you for reading.
I successfully set up the ssh config file to loggin to the AWS.
When I try to do ssh login in my local terminal, it works well, but when I try to do using my VSCode Open-SSH extension, it always fails except the first try.
The output is like this:
[18:38:25.400] Running script with connection command: ssh -T -D 53736 -o ConnectTimeout=15 -F <config> awsserver bash
[18:38:26.521] >
> SessionManagerPlugin is not found. Please refer to SessionManager Documentation here: http://docs.aws.amazon.com/console/systems-manager/session-manager-plugin-not-found
All aws commands are well reached from my terminal environment.
Thank you in advance.
I'm not familiar with the VSCode Open-SSH extension, but appears you are getting a message from Amazon's AWS CLI as if this command was being run:
aws ssm start-session --target i-0d2a6aaaaaaaa61c5
Rather than using ssh, is your extension perhaps configured to use Amazon SSM?
I have an EMR cluster 5.28.1 running in AWS but I forgot to install from python libraries as part of the bootstrap action. Now that the cluster is running, I was simply attempting to add a step via the EMR console. Here are my settings
JAR: s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar
Main class: None
Arguments: s3://xxxx/install_python_libraries.sh
Unfortunately, I get the following error.
Cannot run program "s3://xxxxx/install_python_libraries.sh" (in directory "."): error=2, No such file or directory
I am not sure what I am doing wrong. The shell script looks like this.
#!/bin/bash -xe
# Non-standard and non-Amazon Machine Image Python modules:
sudo pip-3.6 install boto3
sudo pip-3.6 install xmltodict
I also tried this by simply using 'command-runner.jar' but I get the same error. Can you please help me figure out the problem so I do this via the console? I would like to install the libraries on all nodes - master and core.
Thanks
The issue is the xxx.sh files EOL/carriage return type.
In other words, if it is Windows ("\r\n") then it will not work and return the ./ file not found error.
Convert it to unix type ("\n") using something like notepad++ and it will run fine.
(In notepad++ edit>EOL Conversion>Unix(LF) hit save and try again)
I am very new to the Amazon Web Services and have been trying a learn-by-doing approach with them.
In summary I was trying to set up Git with the elastic beanstalk command line interface for my web-app. However, I wanted to use my SSH key-pair to authenticate (aws-access-id, secret) and in my naivety and ignorance, I just supplied this information (the SSH key files) and now I can't get it to work. More specifically stated below.
I have my project directory with Git set up so that it works. I then open the git bash window MINGW64 (I am on Windows 10) and attempt to set up eb.
$ eb init
It then tells me that my credentials are not set up and asks me for aws-access-id and the secret. I had just set up the SSH key-pair and try to enter these files; what's the harm in trying? EB failure, it turns out. Now, the instances seem to run fine still, looking at their status on the AWS console website. However, whatever I type into the bash:
$ eb init
$ eb status
$ eb deploy
$
There is no output. Not even an error. It just silently returns to awaiting a new command from me.
When using the --debug option with these commands, a long list of operations is returned, ending with
botocore.parsers.ResponseParserError: Unable to parse response (no element found: line 1, column 0), invalid XML received:
b''
I thought I would be able to log out or something the like, so that I could enter proper credentials which I messed up from the beginning. I restarted the web-app from the AWS webpage interface and restarted my PC. No success.
Thanks in advance.
EDIT:
I also tried reinstalling awscli and awsebcli:
pip uninstall awsebcli
pip uninstall awscli
pip install awscli
pip install awsebcli --upgrade --user
Problem persists, but now there is one output (previously seen only upon --debug option):
$ eb init
ERROR: ResponseParserError - Unable to parse response (no element found: line 1, column 0), invalid XML received:
b''
$
It sounds like you have replaced your AWS credentials in ~/.aws/credentials and/or ~/.aws/config file(s) with your SSH key. You could manually replace these or execute aws configure if you have the AWS CLI installed.
When running the following command on kube-master (CoreOS):
export KUBERNETES_PROVIDER=aws; wget -q -O - https://get.k8s.io | bash
I get following error:
Can't find aws in PATH, please fix and retry.
I have already set PATH. Can anyopne tell which 'aws' it is searching for? Is it the aws directory in kubernetes repo directory i.e. kubernetes/cluster/aws?
Follow the AWS CLI installation guide and then ensure your PATH is set correctly.
Yes, you are right.
If you set "aws" as KUBERNETES_PROVIDER, Kubernetes will use scripts that reside in kubernetes/cluster/aws. If no KUBERNETES_PROVIDER is set, I believe the default it to rely on gcloud CLI tool.
If you are using Ubuntu OS. run the below command. it will resolve your issue.
apt-get install awscli