I have springboot project in gitlab that I try to deploy to Amazon Elastic Beanstalk environment that I created via Amazon web console.
In gitlab I am using Default shared Runners.
to deploy to AWS I must use 'eb' cli, for this I also need python
I tired to use this in my .gitlab-ci.yml
deploy:
stage: deploy
image: python:3.7.2-alpine3.7
script:
- pip install -r awsebcli
- eb use myenv-env -v
- eb deploy myenv-env -v
But got errors:
Pulling docker image python:3.7.2-alpine3.7 ...
Using docker image sha256:00be2573e9f79754b17954ba7a310a5f70c25b6f5bb78375e27e9e86d874877e for python:3.7.2-alpine3.7 ...
Running on runner-0c303413-project-4590-concurrent-0 via ed06f0845ef4...
Fetching changes...
Removing .m2/
Removing target/
HEAD is now at 5c19020 aws1
Checking out 5c190200 as master...
Skipping Git submodules setup
Checking cache for default...
Successfully extracted cache
Downloading artifacts for build (14719)...
Downloading artifacts from coordinator... ok id=14719 responseStatus=200 OK token=Yy7j2TGy
$ pip install -r awsebcli
Could not open requirements file: [Errno 2] No such file or directory: 'awsebcli'
ERROR: Job failed: exit code 1
Update
when I changed the line into:
pip3 install awsebcli
got:
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe6611ff2e8>: Failed to establish a new connection: [Errno -2] Name does not resolve')': /simple/awsebcli/
Could not find a version that satisfies the requirement awsebcli (from versions: )
No matching distribution found for awsebcli
The problem was proxy.
I added this:
pip3 --proxy=myProxy:myPort install awsebcli --upgrade
Related
These are the error details. Trying to install the awscli through pip. tried wget method as well.
$ pip install awscli
Collecting awscli
Using cached awscli-1.25.53-py3-none-any.whl (3.9 MB)
Collecting PyYAML<5.5,>=3.10
Using cached PyYAML-5.4.1.tar.gz (175 kB)
0 [main] python 2066 child_copy: stack write copy failed, 0xFFFF5480..0x100000000, done 4294923504, windows pid 16392, Win32 error 5
5496 [main] python 2066 dofork: child
2067 - pid 11768, exitval 0x103, errno 11
ERROR: Error [Errno 11] Resource
temporarily unavailable while executing command pip subprocess to install build dependencies
Installing build dependencies ... error
ERROR: Could not install packages due to an OSError: [Errno 11] Resource temporarily unavailable
Any help will be appreciated.
From Installing or updating the latest version of the AWS CLI - AWS Command Line Interface:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
I'm trying to add the following yum repository to my AWS instance:
https://centos.pkgs.org/7/centos-x86_64/
The issue is that there is no repodata/ directory with the required metadata in this source, how can I add this repository without getting the following error:
yum-config-manager --add-repo https://centos.pkgs.org/7/centos-x86_64/
yum install -y katello
https://centos.pkgs.org/7/centos-x86_64/repodata/repomd.xml: [Errno 12] Timeout on
https://centos.pkgs.org/7/centos-x86_64/repodata/repomd.xml: (28, 'Connection timed out
after 5001 milliseconds')
I'm currently missing multiple dependencies like python-rhsm and selinux-policy which both only exist on the centos.pkgs repository.
Your URL is wrong, the following worked for me:
RUN curl http://mirror.centos.org/centos/7/os/x86_64/RPM-GPG-KEY-CentOS-7 -o RPM-GPG-KEY-CentOS-7
RUN rpm --import RPM-GPG-KEY-CentOS-7
RUN yum-config-manager --add-repo='http://mirror.centos.org/centos/7/os/x86_64/'
Note that a lot of their packages will conflict with Amazon's pre-installed packages.
One workaround I've found is to not add that repo, and instead use yum install <direct_rpm_link> for all of my installations. I had to manually resolve some dependencies myself by adding more .rpm links, but at least it worked in the end.
I am attempting to upload a rails app that was recently updated from rails 5.2 to 6 to AWS Elastic Beanstalk. We had someone else working on this, but with the pandemic he had to step away - and now our site is kind of in limbo and I have not been able to update it. I have searched many different variations of my problem but no solutions have worked yet.
The app was working on EB with rails 5.2. I have the app running in 6.0 locally. When I eb deploy I get this error:
MacBook-Pro:app $ eb deploy
Starting environment deployment via CodeCommit
--- Waiting for Application Versions to be pre-processed ---
Finished processing application version app-0e294-200420_110159
2020-04-21 00:22:24 INFO Environment update is starting.
2020-04-21 00:23:07 INFO Deploying new version to instance(s).
2020-04-21 00:27:59 ERROR [Instance: i-0e613ac1fe175f3f6] Command failed on instance. Return code: 1 Output: (TRUNCATED)...-- : Writing /var/app/ondeck/public/assets/application-06fe3df6175ba0def3d0e732489f883d0c09de.css.gz
Webpacker requires Node.js ">=10.13.0" and you are using v6.17.1
Please upgrade Node.js https://nodejs.org/en/download/
Exiting!.
Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/11_asset_compilation.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
2020-04-21 00:27:59 INFO Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2020-04-21 00:28:00 ERROR Unsuccessful command execution on instance id(s) 'i-0e613ac1fe175f3f6'. Aborting the operation.
2020-04-21 00:28:00 ERROR Failed to deploy application.
ERROR: ServiceError - Failed to deploy application.
It was giving me a bundler error before this, that I was able to fix by adding a file into .ebextensions that installs the correct version of bundler. I figured the solution to this would be similar.
This post was close to my problem:
Deploy rails react app with webpacker gem on AWS elastic beanstalk
So I added this file to my .ebextensions based off the selected answer of that:
01_update_note.config
commands:
01_download_nodejs:
command: curl --silent --location https://rpm.nodesource.com/setup_10.x | sudo bash -
02_install_nodejs:
command: yum -y install nodejs
However, it did not appear to do anything, I still get the same error. I tried a couple variations of the file based off a few other blog posts about the issue, but the error remains. Is anyone able to point me in the right direction or offer any insight into the problem? I apologize for not being very familiar with AWS or EB yet, but I will do my best to answer additional questions.
Maybe it is caused by yarn install later.
I try the following scripts and remove yarn install then set RAILS_SKIP_ASSET_COMPILATION=false and it works for me.
commands:
01_install_yarn:
command: "sudo wget https://dl.yarnpkg.com/rpm/yarn.repo -O /etc/yum.repos.d/yarn.repo && curl --silent --location https://rpm.nodesource.com/setup_12.x | sudo bash - && sudo yum install yarn -y"
02_download_nodejs:
command: curl --silent --location https://rpm.nodesource.com/setup_12.x | sudo bash -
03_install_nodejs:
command: yum -y install nodejs
04_install_packages:
command: sudo yum install -y yarn
This is how I did it on Amazon Linux 2:
Create this file in .platform/hooks/prebuild/yarn_config.sh:
#!/usr/bin/env bash
curl --silent --location https://rpm.nodesource.com/setup_16.x | sudo bash -
sudo yum -y install nodejs
sudo wget https://dl.yarnpkg.com/rpm/yarn.repo -O /etc/yum.repos.d/yarn.repo
sudo yum -y install yarn
yarn install
Give it the right permission: chmod +x .platform/hooks/prebuild/yarn_config.sh
And the error is gone, while you assets still compile (unlike with accepted answer)
I reran the startup script using following command:
sudo google_metadata_script_runner --script-type startup
All the yum install commands are failing with following error:
startup-script: INFO startup-script-url: Retrieving key from https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
startup-script: INFO startup-script-url: https://packages.cloud.google.com/yum/repos/cloud-sdk-el7-x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature could not be verified for google-cloud-sdk
startup-script: INFO startup-script-url: Trying other mirror.
startup-script: INFO startup-script-url: One of the configured repositories failed (Google Cloud SDK),
Any idea how I could fix this while instance provisioning or any workaround?
To be honest I have found this on google, not sure if this helps but maybe you can try it out anyway.
1) Disable Caching in yum config /etc/yum.conf:
http_caching=none
2) Delete tmp yum files:
rm -r /var/tmp/yum*
3) restart machines
4) cleanup yum:
yum clean metadata
yum clean all
yum update
I am trying to reproduce the issue on my end. It would be helpful if you could share the information below:
What is the exact OS you are using here.
What happens when you are trying to run the scripts manually after VM starts.
Can you please share the sample script without the confidential information or credentials.
Though I have not tested this, the error can happen due to yum not having enough cached data to continue. And the solution can be found on the public: https://community.cloudera.com/t5/Support-Questions/yum-doesn-t-have-enough-cached-data-to-continue/m-p/220862
I just encountered this same error on a Docker build.
Google Cloud's (latest) repo configuration is as follows:
[google-cloud-cli]
name=Google Cloud CLI
baseurl=https://packages.cloud.google.com/yum/repos/cloud-sdk-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Our Dockerfile yum repo configuration had to be corrected from repo_gpgcheck=1 to repo_gpgcheck=0 and then the error went away.
For me running yum-config-manager --disable google-cloud-sdk solved it. I got this from the error message that I got:
...
4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:
yum-config-manager --disable google-cloud-sdk
or
subscription-manager repos --disable=google-cloud-sdk
In case anyone's facing same issue with apt. Can try --
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
I'm trying to deploy a django project with Elastic Beanstalk cli. I followed the Official Amazon Tutorial as well as this Tutorial on realpython.com.
The Deploymentprocess works fine as long as I stick to a newly created Django Project with just a view dependencies for pip.
When try it with my Django Project The Deployment fails because the requirements.txt is invalid. I created it inside my virutalenv like this: pip freeze > requirements.txt
braintree==3.32.0
cffi==1.9.1
cryptography==1.7
Django==1.10.4
django-allauth==0.29.0
django-betterforms==1.1.4
django-contrib-comments==1.7.3
django-formtools==1.0
django-payments==0.9.6
django-tinymce==2.4.0
enum34==1.1.6
idna==2.1
ipaddress==1.0.17
oauthlib==2.0.1
Pillow==3.4.2
pyasn1==0.1.9
pycparser==2.17
PyJWT==1.4.2
python-openid==2.2.5
requests==2.12.3
requests-oauthlib==0.7.0
six==1.10.0
stripe==1.43.0
suds-jurko==0.6
xmltodict==0.10.2
In my despair I tried to activate the dependencies one by one. The Deployment process is so slow that I gave up after a while. But now I know now that Django==1.10.4, django-allauth==0.29.0, django-betterforms==1.1.4, django-contrib-comments==1.7.3, django-formtools==1.0 and django-tinymce==2.4.0 are not the problem. With django-payments==0.9.6 it fails but its not the only dependency that causes the problem.
This is in my activity.log:
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/opt/python/run/venv/bin/python2.7 -c "import setuptools, tokenize;
__file__='/tmp/pip-build-BF9Oen/cffi/setup.py';
exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
install --record /tmp/pip-BsXWzo-record/install-record.txt --single-version-externally-managed --compile --install-headers
/opt/python/run/venv/include/site/python2.7/cffi" failed with error code 1 in /tmp/pip-build-BF9Oen/cffi
You are using pip version 7.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
2016-12-13 14:49:05,155 ERROR
Error installing dependencies: Command '/opt/python/run/venv/bin/pip install -r /opt/python/ondeck/app/requirements.txt' returned non-zero exit status 1
Traceback (most recent call last):
File "/opt/elasticbeanstalk/hooks/appdeploy/pre/03deploy.py", line 22, in main
install_dependencies()
File "/opt/elasticbeanstalk/hooks/appdeploy/pre/03deploy.py", line 18, in install_dependencies
check_call('%s install -r %s' % (os.path.join(APP_VIRTUAL_ENV, 'bin', 'pip'), requirements_file), shell=True)
File "/usr/lib64/python2.7/subprocess.py", line 541, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command
'/opt/python/run/venv/bin/pip install -r /opt/python/ondeck/app/requirements.txt'
returned non-zero exit status 1 (Executor::NonZeroExitStatus)
I't could have something to do with missing Software. I know how to use the .ebextensions .config files but I don't know whats missing. Is there a descent way to debug this locally?
You seem to have problems with both FFI and Python. You have Cryptography in use but don't probably have all the required libraries Install Python and FFI requirements and OpenSSL dependencies for Cryptography.
$ eb ssh # SSH into your EB environment
$ sudo su # gain root privileges
$ yum install python-devel # or python27-devel
$ yum install libffi-devel # CFFI / FFI requirements
$ yum install openssl-devel # Cryptography requirements
If Pip complains, make sure you have the latest pip installed:
# repeat SSH and root privilege steps
$ source /opt/python/run/venv/bin/activate # activate Python environment
$ pip install --upgrade pip # upgrade to latest pip
To make the changes permanent, append them to your EB configuration. This also works if you do not have direct SSH access.
# .ebextensions/10_packages.config excerpt
packages:
yum:
gcc: []
libffi-devel: []
openssl-devel: []
python-devel: [] # or python27-devel
.ebextensions folder files with the .config suffix are state declarations for the Elastic Beanstalk environment. That is, they are instructions that are run on each Elastic Beanstalk deployment when you run eb deploy or your CI environment deploys code to EB.
Amazon Linux is an RPM based Linux that uses YUM for package management. You will find RHEL / CentOS / OpenSUSE instructions for Python package installation helpful for the Amazon Linux distribution.
There typically isn’t a way to debug these issues locally, because people don’t normally run Amazon linux on their local machines, and EB has a very specific environment setup that is not easy to reproduce (at least it wasn’t for me).
If you want to do it fast, just ssh to one of your EB/EC2 instances and try to install the pip requirements one-by-one by hand. If you want to do it right, you do have to go through this list of requirements one-by-one. (i.e., start with one requirement in the requirements file, deploy, then two requirements, then three requirements, etc.). You may want to start with upgrading pip by adding
pip==9.0.1
At the top of your requirements file and also making sure the following Amazon Linux (if that is what you are using) are installed:
python-devel
libffi
libffi-devel