I need to use Kafka with flask-socketIO for a project. So I decided to use flasfka app that does push/pull to Kafka through http.
Following the instruction, I added FLASFKA_CONFIG path in environment
export FLASFKA_CONFIG=/etc/flasfka.conf.py
But when I run flasfka-serve to start flasfka, I get error
flasfka-serve: command not found
I even added the above export command in my .bash_profile file and restarted the terminal but I am still getting the error.
Looks like the documentation is WRONG.
It says
pip install kafka
instead it should be
pip install flasfka
Related
I'm setting up a new Apache Airflow environment using MWAA with a requirements.txt file. (AirFlow version 2.2.2)
In the requirements.txt file, I am specifically stating a certain version of Amazon provider (apache-airflow-providers-amazon==5.1.0).
After initializing the environment, in the Airflow UI, it says my DAGs are broken due to import errors (ModuleNotFoundError: No module named 'airflow.providers.amazon.aws.sensors.batch')
When opening the scheduler logs, I see that it indeed installed the requested version (5.1.0) and its dependencies, but after looking at the providers, I see that apache-airflow-providers-amazon uses version 2.4.0; no matter what version I state in the requirements.txt file it still shows 2.4.0.
I have updated the environment several times using different versions, but still no luck.
When running aws-mwaa-local-runner and using the same requirements.txt file, everything works as intended; the providers are indeed the ones specified in the requirements.txt file, and my DAGs are running.
Has anyone experienced something like that when setting up an MWAA?
I have a Django project that works with PostgreSQL on Elastic Beanstalk.
I have found the next error when deploying:
Error: pg_config executable not found.
pg_config is required to build psycopg2 from source.
Please add the directory containing pg_config to the $PATH or specify the full executable path with the option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
I followed psycopg2 on elastic beanstalk - can't deploy app to solve it and it worked! BUT after a while Amazon seems to update my virtual env and the error returns so I have to go back and do the same stuff over and over again.
I also have tried configuring the database instance directly from the Elastic Beanstalk panel but nothing changes.
The database is on RDS and I think it's important to say that when I manually install psycopg2 on my instance and re-deploy everything works fine, I even have made a couple of deploys after that and the problems is not there but it is after a while.
I really want to know how to solve it once for all. Thanks in advance.
It is trying to do is build the postgres drivers from source.
You can deal with this several different ways. For example, you can choose to install the drivers from a binary package instead of building them from source.
in your requirements.txt, replace
psycopg2==2.8.5
with
psycopg2-binary==2.8.5
If, you do insist on building it from source when you deploy to Beanstalk, you will need to use platform hooks to pre-install the dependencies that are needed to compile it. The AL2 instances are pretty bare-bone, so you will need to do roughly the following:
add directory structure .platform/hooks/prebuild
in prebuild, create a script, something like '10_install_dependencies.sh'
at the top of the script you will need #!/usr/bin/sh
Use this script to add needed dependencies. For example, for the posgres development libs you will want
sudo yum install -y <yum-packahe-that-you-need>
You may also end up having to install other development libs needed to build psycopg...
I am trying to write to my PostgreSQL database with AWS Lambda using the python2.7 runtime. I care very little about how I do this, so if anyone has a different way that I can understand that works, I'd love to hear it.
The method I'm currently trying is to use psycopg2, as this is the only way I know. In order to do this, I need to upload the psycopg2 module to my environment on AWS Lambda. As per instructions, I've created a directory with my source and psycopg2 using pip install psycopg2 -t ..\my-project, zipped my-project, and uploaded it.
My error message is this from within the AWS Lambda console: Unable to import module 'lambda_function': No module named _psycopg
The code runs on my windows machine. I think the issue is that when I import psycopg2 from my local windows machine, the _psycopg module is being imported from _psycopg.pyd, and .pyd files are windows specific. I may be wrong about this.
I'm really just looking for any way to achieve the desired result described in my first paragraph, but here's a more specific question: How do I tell windows to pip install and compile psycopg2 without using .pyd files? Is this possible? Do I have something completely wrong?
I know the formatting of this question is a little unorthodox, I think I've given all the necessary information, let me know if there's anything else I can provide.
I solved the problem by opening an ubuntu instance on VirtualBox, pip installing the package there, pulling the relevant folders out, and placing them in my-project before zipping and uploading to AWS Lambda.
See these instructions.
I'm deploying an app using Elastic Beanstalk and part of the app has a grunt task that runs "sass", I have sass being installed but it is being installed locally and thus isn't part of the PATH, so the grunt task fails.
I just attempted adding a command to the beanstalk config that does sudo gem install sass but that fails with Command failed on instance. Return code: 1 Output: sudo: sorry, you must have a tty to run sudo.
What would be the best way to get sass into the PATH? There didn't seem to be an easy way to update the PATH / set the .bashrc with elastic beanstalk
Using ebextensions commands is the way to go. You don't need sudo, as the commands run with the necessary privileges.
Also looks like you are using node solution stack (since you mentioned grunt). There may be multiple versions of ruby on your instance. You want to be sure to use the right gem binary so your dependencies are installed in the right location.
There is a ruby installed in /usr/bin and another one in /opt/elasticbeanstalk/lib. The latter is used by Elastic Beanstalk which is not what you want. You want to run the gem binary under /usr/bin.
I'm having trouble installing AWS Elastic Beanstalk command line tool and I don't understand why. I've downloaded the package from AWS and followed the instruction carefully. Following is the installation instruction:
== Installation
Once you have downloaded the CLI package:
1) Unzip this archive to a location of your choosing.
Eb is located in the "eb" directory. The complete CLI reference
for more advanced scenarios can be found in the "api" directory.
To add eb files to your path:
Linux/Mac OS X (Bash shell):
export PATH=$PATH:<path to eb>
Windows:
set PATH=<path to eb>;%PATH%
I'm using Mac OS X so I've used export PATH=$PATH:. For the path to eb, I've just copied the file into the terminal, which resulted export PATH=$PATH:/Users/lydia/Downloads/ElasticBeanstalk/eb/macosx/python2.7/eb. I'm not sure what I'm missing and I can't deploy without downloading eb command line first.
Remove the eb at the end so it's just
/Users/lydia/Downloads/ElasticBeanstalk/eb/macosx/python2.7/
this worked for me although i can only get it to work if i export the CLI into the specific website folder i am working on see my question here https://askubuntu.com/questions/428417/aws-elastic-beanstalk-command-line-tool-setup
Fix that worked for me (if you installed python using brew) is remove python via
brew uninstall --force python
and then install it again from https://www.python.org/downloads/.
Then just follow the instructions from AWS.
You only add directories to your $PATH. Is ~/Downloads/ElasticBeanstalk/eb/macosx/python2.7/eb a directory? Or is it the actual command?