I'm trying to import the BigQueryExecuteQueryOperator on my local Airflow, installed on the M1 macbook.
from airflow.providers.google.cloud.operators.bigquery import BigQueryExecuteQueryOperator
The import returns this error:
| "/Library/Frameworks/Python.framework/V
| ersions/3.10/lib/python3.10/site-packag
| es/pandas/util/_decorators.py", line
| 14, in <module>
| from pandas._libs.properties import
| cache_readonly # noqa:F401
| File
| "/Library/Frameworks/Python.framework/V
| ersions/3.10/lib/python3.10/site-packag
| es/pandas/_libs/__init__.py", line 13,
| in <module>
| from pandas._libs.interval import
| Interval
| ImportError:
| dlopen(/Library/Frameworks/Python.frame
| work/Versions/3.10/lib/python3.10/site-
| packages/pandas/_libs/interval.cpython-
| 310-darwin.so, 0x0002): tried:
| '/Library/Frameworks/Python.framework/V
| ersions/3.10/lib/python3.10/site-packag
| es/pandas/_libs/interval.cpython-310-da
| rwin.so' (mach-o file, but is an
| incompatible architecture (have
| (x86_64), need (arm64e)))
I did some googling and updated my google-cloud-bigquery and grpcio with pip, but the error persists.
Any help will be appreciated. Thanks!
The problem is not with Airflow or GCP provider, it's with pandas and nunpy, you should install the version compatible with the ARM64 architecture (apple M1 for ex), here is an example of a code which I used to solve a similar problem one year ago:
pip3 install Cython
pip3 install --no-binary :all: --no-use-pep517 numpy
pip3 install pandas
pip3 install ib_insync
I work in a M1 Macbook.
For me in my local virtual env, I only have to add the Apache Airflow GCP package :
requirements.txt file
apache-airflow[gcp]==2.3.4
In shell terminal :
pip install -r requirements.txt
To manage Python versions correctly in my machine I use PyEnv : https://github.com/pyenv/pyenv
You can also check the following links that can help with details about environment :
https://github.com/pyenv/pyenv/wiki#suggested-build-environment
https://github.com/pyenv/pyenv/wiki/Common-build-problems
Related
I am having below pipeline in azure devops. I have tested the cdk project on local and it works fine.
When I run the same on azure devops, it gives an error.
pipeline
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '12.x'
displayName: 'Install Node.js'
- script: |
sudo npm install -g aws-cdk
python3 -m pip3 install -r requirements.txt
pip3 install aws_cdk.aws_ec2 aws_cdk.core
cdk synth
displayName: 'npm install and build'
error
2020-09-16T09:20:50.1136286Z Installing collected packages: publication, attrs, cattrs, six, python-dateutil, zipp, importlib-resources, typing-extensions, jsii, aws-cdk.cloud-assembly-schema, aws-cdk.cx-api, aws-cdk.region-info, constructs, aws-cdk.core, aws-cdk.aws-iam, aws-cdk.aws-kms, aws-cdk.aws-cloudwatch, aws-cdk.assets, aws-cdk.aws-events, aws-cdk.aws-s3, aws-cdk.aws-s3-assets, aws-cdk.aws-logs, aws-cdk.aws-ssm, aws-cdk.aws-ec2
2020-09-16T09:20:51.0427755Z Successfully installed attrs-20.2.0 aws-cdk.assets-1.63.0 aws-cdk.aws-cloudwatch-1.63.0 aws-cdk.aws-ec2-1.63.0 aws-cdk.aws-events-1.63.0 aws-cdk.aws-iam-1.63.0 aws-cdk.aws-kms-1.63.0 aws-cdk.aws-logs-1.63.0 aws-cdk.aws-s3-1.63.0 aws-cdk.aws-s3-assets-1.63.0 aws-cdk.aws-ssm-1.63.0 aws-cdk.cloud-assembly-schema-1.63.0 aws-cdk.core-1.63.0 aws-cdk.cx-api-1.63.0 aws-cdk.region-info-1.63.0 cattrs-1.0.0 constructs-3.0.4 importlib-resources-3.0.0 jsii-1.12.0 publication-0.0.3 python-dateutil-2.8.1 six-1.15.0 typing-extensions-3.7.4.3 zipp-3.1.0
2020-09-16T09:20:55.2430033Z Traceback (most recent call last):
2020-09-16T09:20:55.2430990Z File "app.py", line 3, in <module>
2020-09-16T09:20:55.2431651Z from aws_cdk import core
2020-09-16T09:20:55.2432408Z ImportError: No module named aws_cdk
2020-09-16T09:20:55.2468987Z Subprocess exited with error 1
app.py
#!/usr/bin/env python3
from aws_cdk import core
from ec2.ec2_stack import Ec2Stack
app = core.App()
env = core.Environment(account="95XXXXXXXX174", region="us-east-1")
stack=Ec2Stack(app, "ec2",env=env)
app.synth()
Can anyone please guide me here.
Check your cdk.json file, try using python3 app.py instead of python app.py, :
{
"app": "python3 app.py"
}
I met same issue when using python app.py in cdk.json.
I'm trying to run moderngl in Colab. I installed it and ran a virtual display:
!sudo apt-get update --fix-missing && apt-get -qqq install x11-utils > /dev/null
!sudo apt-get update --fix-missing && apt-get -qqq install xvfb > /dev/null
!python3 -m pip install -U -qqq moderngl
!python3 -m pip install -U -qqq moderngl-window
!python3 -m pip install -U -qqq pyvirtualdisplay
from pyvirtualdisplay import Display
display = Display(visible=0, size=(960, 540)).start()
import moderngl
ctx = moderngl.create_standalone_context()
buf = ctx.buffer(b'Hello World!') # allocated on the GPU
buf.read()
b'Hello World!'
It printed as expected, but when I run an example I see the error:
!python3 /content/moderngl/examples/basic_alpha_blending.py --window pyglet
2020-03-28 10:25:48,312 - moderngl_window - INFO - Attempting to load window class: moderngl_window.context.pyglet.Window
Traceback (most recent call last):
File "/content/moderngl/examples/basic_alpha_blending.py", line 74, in <module>
AlphaBlending.run()
File "/content/moderngl/examples/ported/_example.py", line 21, in run
mglw.run_window_config(cls)
File "/usr/local/lib/python3.6/dist-packages/moderngl_window/__init__.py", line 185, in run_window_config
cursor=show_cursor if show_cursor is not None else True,
File "/usr/local/lib/python3.6/dist-packages/moderngl_window/context/pyglet/window.py", line 54, in __init__
config=config,
File "/usr/local/lib/python3.6/dist-packages/pyglet/window/xlib/__init__.py", line 165, in __init__
super(XlibWindow, self).__init__(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pyglet/window/__init__.py", line 588, in __init__
config = screen.get_best_config(config)
File "/usr/local/lib/python3.6/dist-packages/pyglet/canvas/base.py", line 194, in get_best_config
raise window.NoSuchConfigException()
pyglet.window.NoSuchConfigException
I also tried with another virtual display, but the result is the same:
!python3 -m pip install -U -qqq xvfbwrapper
from xvfbwrapper import Xvfb
display = Xvfb(width=960, height=540).start()
pyglet.window.NoSuchConfigException
In Google Colab you can use the EGL backend with moderngl 5.6.
ctx = moderngl.create_context(standalone=True, backend='egl')
print(ctx.info)
Output (partial):
{
'GL_VENDOR': 'NVIDIA Corporation',
'GL_RENDERER': 'Tesla P100-PCIE-16GB/PCIe/SSE2',
'GL_VERSION': '3.3.0 NVIDIA 418.67',
....
}
moderngl creates an OpenGL 3.3 core context by default. If you need a higher context version you can pass in require=430 for OpenGL 4.3 for example. I don't know what these Tesla cards support.
There is a standard example in moderngl for this. It would be able to create the standard RGB triangle: https://github.com/moderngl/moderngl/blob/master/examples/headless_egl.py
The underlying library creating the contexts is glcontext (https://github.com/moderngl/glcontext).
if you are using the moderngl-window package you have to use the headless.Window because pyglet currently is not able to work in headless mode (It might in the future: https://github.com/pyglet/pyglet/issues/51)
If you run into issue make an issue in the moderngl project: https://github.com/moderngl/moderngl or invade their discord server.
I am getting this import error from the scipy package. Please see requirements.txt for versions. This is only happening on Flask and not when I run the same code on ipython. Is there any reason why statsmodels doesn't work on flask but would work on ipython?
File "/env/lib/python3.7/site-packages/statsmodels/api.py", line 9, in
from . import regression
File "/env/lib/python3.7/site-packages/statsmodels/regression/init.py", line 1, in
from .linear_model import yule_walker
File "/env/lib/python3.7/site-packages/statsmodels/regression/linear_model.py", line 39, in
from scipy.linalg import toeplitz
File "/env/lib/python3.7/site-packages/scipy/init.py", line 156, in
from . import fft
File "/env/lib/python3.7/site-packages/scipy/fft/init.py", line 81, in
from ._helper import next_fast_len
File "/env/lib/python3.7/site-packages/scipy/fft/_helper.py", line 4, in
from . import _pocketfft
File "/env/lib/python3.7/site-packages/scipy/fft/_pocketfft/init.py", line 3, in
from .basic import *
File "/env/lib/python3.7/site-packages/scipy/fft/_pocketfft/basic.py", line 8, in
from . import pypocketfft as pfft
ImportError: SystemExit: 1
*This happens only in Flask. When i use statsmodels
I'm running it in flask and here are my complete list of packages:
attrs==19.3.0
autopep8==1.5
backcall==0.1.0
bleach==3.1.0
cachetools==4.0.0
cattrs==1.0.0
certifi==2019.11.28
chardet==3.0.4
Click==7.0
cloudstorage==0.10.0
cycler==0.10.0
decorator==4.4.1
defusedxml==0.6.0
entrypoints==0.3
Flask==1.1.1
Flask-Assets==2.0
Flask-CacheBuster==1.0.0
Flask-Cors==3.0.8
Flask-SQLAlchemy==2.4.1
flask-talisman==0.7.0
google-api-core==1.16.0
google-auth==1.11.0
google-cloud==0.34.0
google-cloud-core==1.2.0
google-cloud-storage==1.25.0
google-cloud-tasks==1.3.0
google-resumable-media==0.5.0
googleapis-common-protos==1.51.0
grpc-google-iam-v1==0.12.3
grpcio==1.26.0
idna==2.8
importlib-metadata==1.5.0
inflect==3.0.2
inflection==0.3.1
ipykernel==5.1.4
ipython==7.11.1
ipython-genutils==0.2.0
ipywidgets==7.5.1
itsdangerous==1.1.0
jedi==0.16.0
Jinja2==2.11.0
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==5.3.4
jupyter-console==6.1.0
jupyter-contrib-core==0.3.3
jupyter-contrib-nbextensions==0.5.1
jupyter-core==4.6.1
jupyter-highlight-selected-word==0.2.0
jupyter-latex-envs==1.4.6
jupyter-nbextensions-configurator==0.4.1
kiwisolver==1.1.0
looker-sdk==0.1.3b6
lxml==4.5.0
MarkupSafe==1.1.1
matplotlib==3.1.2
mistune==0.8.4
natural==0.2.0
nbconvert==5.6.1
nbformat==5.0.4
nltk==3.4.5
notebook==6.0.3
numpy==1.18.1
pandas==0.25.3
pandas-datareader==0.8.1
pandocfilters==1.4.2
parso==0.6.0
patsy==0.5.1
pexpect==4.8.0
pg8000==1.13.2
pickleshare==0.7.5
prometheus-client==0.7.1
prompt-toolkit==3.0.3
protobuf==3.11.2
psycopg2==2.8.4
ptyprocess==0.6.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.5.0
Pygments==2.5.2
pyparsing==2.4.6
pyrsistent==0.15.7
python-dateutil==2.8.1
python-magic==0.4.15
pytz==2019.3
PyYAML==5.3
pyzmq==18.1.1
qtconsole==4.6.0
requests==2.22.0
rsa==4.0
scipy==1.4.1
scramp==1.1.0
seasonal==0.3.1
Send2Trash==1.5.0
simple-salesforce==0.74.3
simplenlg==0.2.0
six==1.14.0
SQLAlchemy==1.3.13
statsmodels==0.11.0
stripe==2.42.0
terminado==0.8.3
testpath==0.4.4
tornado==6.0.3
traitlets==4.3.3
tzlocal==2.0.0
urllib3==1.25.8
wcwidth==0.1.8
webapp2==2.5.2
webassets==2.0
webencodings==0.5.1
Werkzeug==0.16.1
widgetsnbextension==3.5.1
zipp==2.1.0
no error in ipython, but i get an error when i run it in flask
Checkout this issues for more detail.
Turns out that statsmodels is dependent upon several packages being installed before it so that it can key on them to compile its own modules. Although I don't exactly know what causes this particular conflict with flask, I have faced this on python3.7 before and here is what worked for me:
If you need to clean out what you already have, you can uninstall with the following command:
pip3 uninstall statsmodels
Now install Flask
pip3 install Flask
Install the individual dependencies of statmodels first:
pip3 install numpy scipy patsy pandas
Now install statmodels:
pip3 install statsmodels
Important!!!
Make sure you are running a fresh virtual environment. To open a new one in linxux system:
python3 -m venv venv
source venv/bin/activate
Now install the dependencies as instructed above. Let me know if that fixed your problem. If not, you might need a better dependency resolver than pip. Poetry is one of them.
Just restart the server and the problem is solved!
for my distutils package the yaml file along with python files is not getting copied
setup.py
from distutils.core import setup
files = ["*.yaml", "package/*"]
setup(name = "mypackage",
version = "0.1",
description = "description",
author = "Ciasto",
author_email = "me#email.com",
packages = ['package'],
package_data = {'package' : files },
scripts = ["scripts/runner"],
)
this is the project directory structure:
$ tree package/
|____
| |______init__.py
| |____command.py
| |____constants.py
| |____controller.py
| |____utils.py
| |____model.py
| |____products.yaml
package_data is used to add package's data to eggs (dropped in favor) and wheels (not with distutils). You're probably generating source distribution (sdist).
For sdist you need file MANIFEST.in (create it besides setup.py). In your case it should be enough to have one line in it:
include package/*.yaml
See the docs at https://docs.python.org/3/distutils/sourcedist.html#specifying-the-files-to-distribute
and
https://packaging.python.org/guides/using-manifest-in/#using-manifest-in
If you're not going to create wheels you can safely remove files and package_data from setup.py.
I tried several things to get my app working on heroku, but now I'm out of ideas. I can install my project on heroku's rep, but I get a 500 error code. My application works very well using virtualenv on my machine after I followed the steps described on heroku documentation for django.
When I do my "git push heroku master" and try in browser, I get the following error:
2013-07-07T15:39:11.170514+00:00 app[web.1]: ImportError: No module named apps.base
2013-07-07T15:39:11.170059+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/django/utils/dateformat.py", line 35, in format
2013-07-07T15:39:11.170202+00:00 app[web.1]: app = import_module(appname)
2013-07-07T15:39:11.170202+00:00 app[web.1]: default_translation = _fetch(settings.LANGUAGE_CODE)
2013-07-07T15:39:11.170202+00:00 app[web.1]: _default = translation(settings.LANGUAGE_CODE)
I tought it was caused by a directory structure that wasn't supported on heroku, so I adjusted it from the default one that was created with django startproject command.
Here is my new file structure. I adjusted the import reference everywhere and as I said, I works pefctly in local:
manage.py
Procfile
requirements.txt
vielfaltig
|____apps
| |____base
| | |____models.py
| | |____templates
| | |____tests.py
| | |____urls.py
| | |____views.py
| |____projects
| | |____admin.py
| | |____models.py
| | |____templates
| | |____templatetags
| | |____tests.py
| | |____translation.py
| | |____urls.py
| | |____views.py
|____locale
|____media
|____settings.py
|____static
|____urls.py
|____vielfaltig.db
|____wsgi.py
As you notice, I have 2 apps (base and projects). In the code, I import them using "vielfaltig.apps.base" for example. I changed this everywhere. I had this error before and I changed the directory structure according to what I read when I googled the error. I also tried to put everything in the root directory (along with the requirements.txt and procfile). I don't know why it keeps telling me an ImportError for "apps.base" while I reference the app using "vielfaltif.apps.base" everywhere... ?
Does anyone have an idea? I will paste my settings.py if needed. For now I think it would just take a lot of space.
Thank you very much for any help !
I guess it will work with import vielfaltig.apps.base instead of apps.base. It can be fixed if you insert path vielfaltig to python system path in your settings.py
import sys
sys.path.append('vielfaltig')