I have written a dag which uses mongoexport command in BashOperator. By default mongoexport package is not installed in composer. I will need to install it using below command:
sudo apt install mongo-tools
We can directly install PyPi packages in composer as there is direct option available in gcp console for that.
But how can i install apt package in all nodes of composer? also that package should be effective in case of autoscaling.
Related
I am trying to deploy a Flask/ML app to Google Cloud but encountering this issue;
When I try to run gcloud builds submit --tag gcr.io/project-name/index with a requirements.txt file, it throws CMake must be installed to build dlib. Since now, I didn't have any problems working with dlib (I have cmake and build-essential already installed) and when I connect Google Cloud Shell via ssh, I can see both build-essential and cmake is installed and up to date on there too.
I tried to build on Ubuntu and Windows machines, same error consists.
Solved! CMake needs to be built in Dockerfile, not locally or in cloud shell.
Adding RUN apt-get update && apt-get -y install cmake before RUN pip install -r requirements.txt in Dockerfile resolved the issue.
I installed on CentOS 7 awscli using this command:
yum install awscli
But it turned out that it is old version of awscli:
awscli.noarch 1.14.28-5.el7_5.1
How could I update or install using yum on CentOS 7 to get awscli 1.18?
AWS documentation here clearly mentions as below. Looks like it is not recommended to use the yum appraoch.
Although the awscli package is available in repositories for other package managers such as apt and yum, these are not produced, managed, or supported by AWS. We recommend that you install the AWS CLI from only the official AWS distribution points, as documented in this guide.
after installing the gcp package to my airflow(1.10.9) set-up, i tried to call on the GSheetHook
https://airflow.readthedocs.io/en/latest/_api/airflow/providers/google/suite/hooks/sheets/index.html
but i get an error No module named 'airflow.providers'.
looking into the installed python packages for airflow, i do not find the providers package.
is the gcp airflow packagge working or am i missingg a step before i am able to use it?
EDIT: I have installed the gcp package using the pip installer: pip install apache-airflow[gcp]
and here's the list of the installed packages
The "providers" package is only available in Airflow Master. We plan to release each provider as a separate packages as "backport packages", most likely in a week or two from today.
PR to do that: https://github.com/apache/airflow/pull/8807
You should be checking https://airflow.apache.org/docs/1.10.9/ for Airflow 1.10.9 docs. You are looking at the docs for "latest" which is for Master.
It looks backport packages (providers) can now be installed in v1.10* using with following pip command $pip install apache-airflow-backport-providers-PACKAGE-NAME
How to install FFMPEG in Google Cloud Shell. is that possible?
screenshot of error
sorry for my bad language
You can install FFmpeg on Cloud Shell using the following commands. However, your Cloud Shell machine is ephemeral and no system-wide change will persist beyond session end.
sudo add-apt-repository ppa:mc3man/trusty-media
sudo apt-get update
sudo apt-get install ffmpeg
Customize Cloud Shell
Using the command line, I have installed PHP 7.0 on the Amazon Linux AMI.
However, I am unable to install the PHP Zip Module using the command
sudo yum install php7.0-zip
I get the response
No package php7.0.zip available.
I am excited to start using PHP7 on all my PHP websites, but am a little confused why I am unable to install this particular module. Any help would be much appreciated!
Welp,
After trying several variations of the same command...
sudo yum install php7.0-zip
sudo yum install php7-zip
THIS ONE WORKS
sudo yum install php70-zip