I'm setting up a new Apache Airflow environment using MWAA with a requirements.txt file. (AirFlow version 2.2.2)
In the requirements.txt file, I am specifically stating a certain version of Amazon provider (apache-airflow-providers-amazon==5.1.0).
After initializing the environment, in the Airflow UI, it says my DAGs are broken due to import errors (ModuleNotFoundError: No module named 'airflow.providers.amazon.aws.sensors.batch')
When opening the scheduler logs, I see that it indeed installed the requested version (5.1.0) and its dependencies, but after looking at the providers, I see that apache-airflow-providers-amazon uses version 2.4.0; no matter what version I state in the requirements.txt file it still shows 2.4.0.
I have updated the environment several times using different versions, but still no luck.
When running aws-mwaa-local-runner and using the same requirements.txt file, everything works as intended; the providers are indeed the ones specified in the requirements.txt file, and my DAGs are running.
Has anyone experienced something like that when setting up an MWAA?
Related
this last week I have been trying to upload a flask app using AWS Beanstalk.
The main problem for me was loading a very heavy library as part of the bundle (there is a 500mb limit for uploading the bundle code).
Instead, I tried to use requirements.txt file so it would download the library directly to the server.
Unfortunately, every time I tried to include the library name in the requirements file, it failed to load it (torch library).
on pythonanywhere server there is a console which allows you to access the virtual environment and simply type
pip install torch
which was very useful and comfortable.
I am looking for something similar in AWS beanstalk, so that I could install the library directly instead of relying on the requirements.txt file.
I have been at it for a few days now and can't make any progress.
your help would be much appreciated.
another question,
is it possible to load the venv to Amazon-S3 and then access the folder from the beanstalk environment?
Its not a good practice to "manually" install your dependencies or configure your EB env from inside. This is only useful for testing and debugging purposes. Thus keep that it mind.
To get your venv, you have to ssh to your EB instance using regular ssh or web-based clients available in AWS EC2 console when you locate your EB EC2 instance. Session manager should work out-of-the-box to enable you to login to the instance.
When you login to the instance, then to activate your venv, you do:
# start bash
bash
# source venv
source /var/app/venv/staging-*/bin/activate
This only started happening tonight, and even after reverting my totally not-npm related changes it's still happening.
I've got an AWS Elastic Beanstalk setup here where I'm calling eb deploy to deploy a KeystoneJS cms application. As part of the deployment it runs npm install, and I've got a custom fork/branch of the keystone github repo that it's supposed to install. And it does! But for some inexplicable reason /lib/core/ in the Keystone repo is just... not there. I get errors complaining about those missing files, and sure enough the entire folder is not present. They are just not npm installed, despite the rest of the Keystone repo being installed just fine.
I can't reproduce this locally. I'll run npm install, it adds that folder. I'll do npm install <my-fork>, it adds the folder. Every combination locally works just fine, and every deployment I've done to EBS in the PAST has worked just fine. Only tonight has this folder stopped showing up in my installations.
Is it a problem with Elastic Beanstalk? Is it a problem with npm? I've made sure to sync my local npm version (6.8.0) with the EB one, no difference. I've checked to make sure I don't have any .ebignore or .npmignore or .gitignore that might somehow be blocking the core folder, nothing. Unless there's one secretly controlling the temp folder that gets first installed to? I don't know why this would suddenly be an issue though, when it wasn't a couple weeks ago.
Anyone experienced anything like this?
[Edit] For some additional details, changing the keystone version in my package.json to just keystone: "4.0.0" gets me those core files fine. If I install directly from the associated keystone repo, keystone: "keystonejs/keystone", they aren't there. This is again just on the eb install tho, the core files show up for both if I do them locally. But on eb when I install from a git url, which I need to for my specific fork/branch, I see this issue.
Well, I figured it out!
https://npm.community/t/npm-pack-leaving-out-files-6-8-0-only/5382
Someone broke npm 6.8.0. Let my tale be a cautionary one, don't have your deployment scripts set to auto-update npm to the latest version.
I'm currently developing a web application (Django 2.0) application.
My app will be deployed on IBM Cloud (Cloud Foundry) using python build-pack.
One of my requirements is to install blender.
Everything else is very well, but for blender installation.
What I've tried so far was:
I tried access my app using SSH connection, but surely I don't have root access to apt-get install blender!!
And tried to include blender in packages.json file and push that file using cf push my-app.
But nothing worked for me.
In another shorter question: what is the main approach in Cloud Foundry Apps to install packages like when we use apt-get install in Ubuntu / Debian.
Please correct me if I did anything wrong, or guide me with headlines to solve this problem!!
I see a couple options for you to install packages if they cannot be installed using the regular requirements file (which is the preferred way):
Download the relevant libraries and put them in subfolders of the app before pushing it. The libraries will be uploaded. That is how I would do it.
Once you have an SSH connection, use secure copy (scp) to upload the files and place them in the subfolders where they are expected.
Regarding Blender, the question is what you need in addition to having the code copied over. Does it need a running daemon? Are there more dependencies? You would need to share more information about your specific app to answer that. Maybe, packaging everything as one or more containers and run it on Kubernetes or a combination of Cloud Foundry and Kubernetes is a better way.
I need to use Kafka with flask-socketIO for a project. So I decided to use flasfka app that does push/pull to Kafka through http.
Following the instruction, I added FLASFKA_CONFIG path in environment
export FLASFKA_CONFIG=/etc/flasfka.conf.py
But when I run flasfka-serve to start flasfka, I get error
flasfka-serve: command not found
I even added the above export command in my .bash_profile file and restarted the terminal but I am still getting the error.
Looks like the documentation is WRONG.
It says
pip install kafka
instead it should be
pip install flasfka
Running Django 1.6 and Analytical 0.16.0
I have the following in my settings.py
GOOGLE_ANALYTICS_PROPERTY_ID = env_var('GOOGLE_ANALYTICS_PROPERTY_ID')
GOOGLE_ANALYTICS_DISPLAY_ADVERTISING = True
and the google analytics code shows up as expected when I run the site locally and on the staging server (ie. running the doubleclick dc.js analytics script), however when running on production it still shows the default Google Analytics ga.js script.
It isn't affected by DEBUG being on or off and as I can tell the settings and env are the same on production and staging servers (both runnning on Heroku). Can anyone offer an explanation of why this might be the case?
edit: SOLVED. Turns out I was still running Analytical 0.15.0 on the production server. I had wrongly assumed that heroku automatically installed the latest version if the version wasn't specified in the pip requirements.
Check that Heroku is running the same versions of each program:
heroku pip freeze
It turns out it was still running an old version of django-analytical as the version number wasn't specified in the pip requirements file. Heroku won't upgrade an existing program unless explicitly specified. Changing the requirements.txt to the following solved it.
django-analytical==0.16.0