How to run moderngl in Colab? - opengl

I'm trying to run moderngl in Colab. I installed it and ran a virtual display:
!sudo apt-get update --fix-missing && apt-get -qqq install x11-utils > /dev/null
!sudo apt-get update --fix-missing && apt-get -qqq install xvfb > /dev/null
!python3 -m pip install -U -qqq moderngl
!python3 -m pip install -U -qqq moderngl-window
!python3 -m pip install -U -qqq pyvirtualdisplay
from pyvirtualdisplay import Display
display = Display(visible=0, size=(960, 540)).start()
import moderngl
ctx = moderngl.create_standalone_context()
buf = ctx.buffer(b'Hello World!') # allocated on the GPU
buf.read()
b'Hello World!'
It printed as expected, but when I run an example I see the error:
!python3 /content/moderngl/examples/basic_alpha_blending.py --window pyglet
2020-03-28 10:25:48,312 - moderngl_window - INFO - Attempting to load window class: moderngl_window.context.pyglet.Window
Traceback (most recent call last):
File "/content/moderngl/examples/basic_alpha_blending.py", line 74, in <module>
AlphaBlending.run()
File "/content/moderngl/examples/ported/_example.py", line 21, in run
mglw.run_window_config(cls)
File "/usr/local/lib/python3.6/dist-packages/moderngl_window/__init__.py", line 185, in run_window_config
cursor=show_cursor if show_cursor is not None else True,
File "/usr/local/lib/python3.6/dist-packages/moderngl_window/context/pyglet/window.py", line 54, in __init__
config=config,
File "/usr/local/lib/python3.6/dist-packages/pyglet/window/xlib/__init__.py", line 165, in __init__
super(XlibWindow, self).__init__(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pyglet/window/__init__.py", line 588, in __init__
config = screen.get_best_config(config)
File "/usr/local/lib/python3.6/dist-packages/pyglet/canvas/base.py", line 194, in get_best_config
raise window.NoSuchConfigException()
pyglet.window.NoSuchConfigException
I also tried with another virtual display, but the result is the same:
!python3 -m pip install -U -qqq xvfbwrapper
from xvfbwrapper import Xvfb
display = Xvfb(width=960, height=540).start()
pyglet.window.NoSuchConfigException

In Google Colab you can use the EGL backend with moderngl 5.6.
ctx = moderngl.create_context(standalone=True, backend='egl')
print(ctx.info)
Output (partial):
{
'GL_VENDOR': 'NVIDIA Corporation',
'GL_RENDERER': 'Tesla P100-PCIE-16GB/PCIe/SSE2',
'GL_VERSION': '3.3.0 NVIDIA 418.67',
....
}
moderngl creates an OpenGL 3.3 core context by default. If you need a higher context version you can pass in require=430 for OpenGL 4.3 for example. I don't know what these Tesla cards support.
There is a standard example in moderngl for this. It would be able to create the standard RGB triangle: https://github.com/moderngl/moderngl/blob/master/examples/headless_egl.py
The underlying library creating the contexts is glcontext (https://github.com/moderngl/glcontext).
if you are using the moderngl-window package you have to use the headless.Window because pyglet currently is not able to work in headless mode (It might in the future: https://github.com/pyglet/pyglet/issues/51)
If you run into issue make an issue in the moderngl project: https://github.com/moderngl/moderngl or invade their discord server.

Related

Why is statsmodels package not working on Flask but is on ipython?

I am getting this import error from the scipy package. Please see requirements.txt for versions. This is only happening on Flask and not when I run the same code on ipython. Is there any reason why statsmodels doesn't work on flask but would work on ipython?
File "/env/lib/python3.7/site-packages/statsmodels/api.py", line 9, in
from . import regression
File "/env/lib/python3.7/site-packages/statsmodels/regression/init.py", line 1, in
from .linear_model import yule_walker
File "/env/lib/python3.7/site-packages/statsmodels/regression/linear_model.py", line 39, in
from scipy.linalg import toeplitz
File "/env/lib/python3.7/site-packages/scipy/init.py", line 156, in
from . import fft
File "/env/lib/python3.7/site-packages/scipy/fft/init.py", line 81, in
from ._helper import next_fast_len
File "/env/lib/python3.7/site-packages/scipy/fft/_helper.py", line 4, in
from . import _pocketfft
File "/env/lib/python3.7/site-packages/scipy/fft/_pocketfft/init.py", line 3, in
from .basic import *
File "/env/lib/python3.7/site-packages/scipy/fft/_pocketfft/basic.py", line 8, in
from . import pypocketfft as pfft
ImportError: SystemExit: 1
*This happens only in Flask. When i use statsmodels
I'm running it in flask and here are my complete list of packages:
attrs==19.3.0
autopep8==1.5
backcall==0.1.0
bleach==3.1.0
cachetools==4.0.0
cattrs==1.0.0
certifi==2019.11.28
chardet==3.0.4
Click==7.0
cloudstorage==0.10.0
cycler==0.10.0
decorator==4.4.1
defusedxml==0.6.0
entrypoints==0.3
Flask==1.1.1
Flask-Assets==2.0
Flask-CacheBuster==1.0.0
Flask-Cors==3.0.8
Flask-SQLAlchemy==2.4.1
flask-talisman==0.7.0
google-api-core==1.16.0
google-auth==1.11.0
google-cloud==0.34.0
google-cloud-core==1.2.0
google-cloud-storage==1.25.0
google-cloud-tasks==1.3.0
google-resumable-media==0.5.0
googleapis-common-protos==1.51.0
grpc-google-iam-v1==0.12.3
grpcio==1.26.0
idna==2.8
importlib-metadata==1.5.0
inflect==3.0.2
inflection==0.3.1
ipykernel==5.1.4
ipython==7.11.1
ipython-genutils==0.2.0
ipywidgets==7.5.1
itsdangerous==1.1.0
jedi==0.16.0
Jinja2==2.11.0
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==5.3.4
jupyter-console==6.1.0
jupyter-contrib-core==0.3.3
jupyter-contrib-nbextensions==0.5.1
jupyter-core==4.6.1
jupyter-highlight-selected-word==0.2.0
jupyter-latex-envs==1.4.6
jupyter-nbextensions-configurator==0.4.1
kiwisolver==1.1.0
looker-sdk==0.1.3b6
lxml==4.5.0
MarkupSafe==1.1.1
matplotlib==3.1.2
mistune==0.8.4
natural==0.2.0
nbconvert==5.6.1
nbformat==5.0.4
nltk==3.4.5
notebook==6.0.3
numpy==1.18.1
pandas==0.25.3
pandas-datareader==0.8.1
pandocfilters==1.4.2
parso==0.6.0
patsy==0.5.1
pexpect==4.8.0
pg8000==1.13.2
pickleshare==0.7.5
prometheus-client==0.7.1
prompt-toolkit==3.0.3
protobuf==3.11.2
psycopg2==2.8.4
ptyprocess==0.6.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.5.0
Pygments==2.5.2
pyparsing==2.4.6
pyrsistent==0.15.7
python-dateutil==2.8.1
python-magic==0.4.15
pytz==2019.3
PyYAML==5.3
pyzmq==18.1.1
qtconsole==4.6.0
requests==2.22.0
rsa==4.0
scipy==1.4.1
scramp==1.1.0
seasonal==0.3.1
Send2Trash==1.5.0
simple-salesforce==0.74.3
simplenlg==0.2.0
six==1.14.0
SQLAlchemy==1.3.13
statsmodels==0.11.0
stripe==2.42.0
terminado==0.8.3
testpath==0.4.4
tornado==6.0.3
traitlets==4.3.3
tzlocal==2.0.0
urllib3==1.25.8
wcwidth==0.1.8
webapp2==2.5.2
webassets==2.0
webencodings==0.5.1
Werkzeug==0.16.1
widgetsnbextension==3.5.1
zipp==2.1.0
no error in ipython, but i get an error when i run it in flask
Checkout this issues for more detail.
Turns out that statsmodels is dependent upon several packages being installed before it so that it can key on them to compile its own modules. Although I don't exactly know what causes this particular conflict with flask, I have faced this on python3.7 before and here is what worked for me:
If you need to clean out what you already have, you can uninstall with the following command:
pip3 uninstall statsmodels
Now install Flask
pip3 install Flask
Install the individual dependencies of statmodels first:
pip3 install numpy scipy patsy pandas
Now install statmodels:
pip3 install statsmodels
Important!!!
Make sure you are running a fresh virtual environment. To open a new one in linxux system:
python3 -m venv venv
source venv/bin/activate
Now install the dependencies as instructed above. Let me know if that fixed your problem. If not, you might need a better dependency resolver than pip. Poetry is one of them.
Just restart the server and the problem is solved!

Unable to run chromedriver using selenium in Python in aws cloud9

After I install selenium, chrome, and chromedriver to aws cloud9, I tried to execute chromedriver. But PATH error occurred.
I checked chromedriver path is correct. And I don't know why this error happened.
aws:~/environment/seleniumTest $ ll /home/ec2-user/environment/seleniumTest/chromedriver
-rwxr-xr-x 1 ec2-user ec2-user 8496784 Sep 13 2018 /home/ec2-user/environment/seleniumTest/chromedriver
Are there any solutions to solve this problem?
My steps is as below,
1.install selenium
pip install selenium -t ./
2.install chrome
curl https://intoli.com/install-google-chrome.sh | bash
3.download chromedriver
curl -SL https://chromedriver.storage.googleapis.com/2.42/chromedriver_linux64.zip > chromedriver.zip
unzip chromedriver.zip
4.Execute Python code as below
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
def lambda_handler(event, context):
options = Options()
options.add_argument('--headless')
driver = webdriver.Chrome('/home/ec2-use/environment/seleniumTest/chromedriver', chrome_options=options)
return(0)
I got "'chromedriver' executable needs to be in PATH" error.
Response
{
"errorMessage": "Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home\n",
"errorType": "WebDriverException",
"stackTrace": [
[
"/var/task/seleniumTest/lambda_function.py",
9,
"lambda_handler",
"driver = webdriver.Chrome('/home/ec2-use/environment/seleniumTest/chromedriver', chrome_options=options)"
],
[
"/var/task/selenium/webdriver/chrome/webdriver.py",
73,
"__init__",
"self.service.start()"
],
[
"/var/task/selenium/webdriver/common/service.py",
83,
"start",
"os.path.basename(self.path), self.start_error_message)"
]
]
}
Function Logs
Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
: WebDriverException
Traceback (most recent call last):
File "/var/task/seleniumTest/lambda_function.py", line 9, in lambda_handler
driver = webdriver.Chrome('/home/ec2-use/environment/seleniumTest/chromedriver', chrome_options=options)
File "/var/task/selenium/webdriver/chrome/webdriver.py", line 73, in __init__
self.service.start()
File "/var/task/selenium/webdriver/common/service.py", line 83, in start
os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
I'm not sure you're specifying the executable_path for Chromedriver properly, I think the below should work:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
def lambda_handler(event, context):
options = Options()
options.add_argument('--headless')
driver = webdriver.Chrome(executable_path='/home/ec2-user/environment/seleniumTest/chromedriver', chrome_options=options)
return(0)

Import sklearn fails - conflict with numpy version?

I'm trying to import sklearn module in the interactive python environment, but I get this error message:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Python/2.7/site-packages/sklearn/__init__.py", line 57, in <module>
from .base import clone
File "/Library/Python/2.7/site-packages/sklearn/base.py", line 11, in <module>
from .utils.fixes import signature
File "/Library/Python/2.7/site-packages/sklearn/utils/__init__.py", line 10, in <module>
from .murmurhash import murmurhash3_32
File "numpy.pxd", line 155, in init sklearn.utils.murmurhash (sklearn/utils/murmurhash.c:5029)
ValueError: numpy.dtype has the wrong size, try recompiling
I read somewhere that this might be a problem derived by the fact that scikit-learn is compiled against an older version of numpy, but I tried to uninstall and reinstall both numpy and scikit-learn (and scipy to be sure) and the problem persists.
These are my python modules:
giacomopoli$ pip freeze
Warning: cannot find svn location for setuptools===0.6c12dev-r88846
altgraph==0.10.1
appnope==0.1.0
attrs==15.2.0
backports-abc==0.4
backports.ssl-match-hostname==3.5.0.1
bdist-mpkg==0.4.4
beautifulsoup4==4.4.1
bonjour-py==0.3
certifi==2016.2.28
cssselect==0.9.1
decorator==4.0.9
functools32==3.2.3.post2
gnureadline==6.3.3
ipykernel==4.3.1
ipython==4.1.2
ipython-genutils==0.1.0
ipywidgets==4.1.1
Jinja2==2.8
jsonschema==2.5.1
jupyter==1.0.0
jupyter-client==4.1.1
jupyter-console==4.1.1
jupyter-core==4.0.6
lxml==3.5.0
macholib==1.5
MarkupSafe==0.23
matplotlib==1.1.1
mistune==0.7.2
modulegraph==0.10.1
nbconvert==4.1.0
nbformat==4.0.1
notebook==4.1.0
numpy==1.10.4
path.py==8.1.2
pexpect==4.0.1
pickleshare==0.6
protobuf==3.0.0b2
ptyprocess==0.5.1
py2app==0.7.1
pyasn1==0.1.9
pyasn1-modules==0.0.8
Pygments==2.1.3
PyInstaller==3.1.1
pyobjc-core==2.3.2a0
pyobjc-framework-AddressBook==2.3.2a0
pyobjc-framework-AppleScriptKit==2.3.2a0
pyobjc-framework-AppleScriptObjC==2.3.2a0
pyobjc-framework-Automator==2.3.2a0
pyobjc-framework-CFNetwork==2.3.2a0
pyobjc-framework-Cocoa==2.3.2a0
pyobjc-framework-Collaboration==2.3.2a0
pyobjc-framework-CoreData==2.3.2a0
pyobjc-framework-CoreLocation==2.3.2a0
pyobjc-framework-CoreText==2.3.2a0
pyobjc-framework-DictionaryServices==2.3.2a0
pyobjc-framework-ExceptionHandling==2.3.2a0
pyobjc-framework-FSEvents==2.3.2a0
pyobjc-framework-InputMethodKit==2.3.2a0
pyobjc-framework-InstallerPlugins==2.3.2a0
pyobjc-framework-InstantMessage==2.3.2a0
pyobjc-framework-InterfaceBuilderKit==2.3.2a0
pyobjc-framework-LatentSemanticMapping==2.3.2a0
pyobjc-framework-LaunchServices==2.3.2a0
pyobjc-framework-Message==2.3.2a0
pyobjc-framework-OpenDirectory==2.3.2a0
pyobjc-framework-PreferencePanes==2.3.2a0
pyobjc-framework-PubSub==2.3.2a0
pyobjc-framework-QTKit==2.3.2a0
pyobjc-framework-Quartz==2.3.2a0
pyobjc-framework-ScreenSaver==2.3.2a0
pyobjc-framework-ScriptingBridge==2.3.2a0
pyobjc-framework-SearchKit==2.3.2a0
pyobjc-framework-ServiceManagement==2.3.2a0
pyobjc-framework-SyncServices==2.3.2a0
pyobjc-framework-SystemConfiguration==2.3.2a0
pyobjc-framework-WebKit==2.3.2a0
pyobjc-framework-XgridFoundation==2.3.2a0
pyOpenSSL==0.13
PyRSS2Gen==1.0.0
python-dateutil==1.5
pytz===2012d
pyzmq==15.2.0
qtconsole==4.2.0
queuelib==1.4.2
scikit-learn==0.17.1
scipy==0.17.0
Scrapy==1.0.5
selenium==2.52.0
service-identity==16.0.0
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.10.0
tensorflow==0.7.1
terminado==0.6
tornado==4.3
traitlets==4.1.0
Twisted==12.2.0
w3lib==1.13.0
xattr==0.6.4
zope.interface==3.8.0
I've Python 2.7 and OSX 10.9.5.
What do I have to do in order to get things work?
I figured it out. I had multiple versions of python installed (2.3, 2.5, 2.6, 2.7) and multiple version of Numpy. The one used in the interactive shell was numpy 1.6. I erased them all and installed the new one with numpy and now it works.

Error in TensorFlow program

I am learning TensorFlow and I stumble upon this example code for creating simple multi-layer sigmoid network. The program in the link is for MNIST database and hand written digit classification.
I want to train a network for regression task. I have 30 inputs(float) which is used to predict one output(float). So I tweaked the code to change the task from classification to regression.
My problem is that I'm getting an error in tf.Session.run(). The code and the error log is given below.
import test2
import tensorflow as tf
feed_input = test2.read_data_sets()
learning_rate = 0.001
training_epochs = 100
batch_size = 1716
display_step = 1
n_hidden_1 = 256
n_hidden_2 = 256
n_hidden_3 = 256
n_input = 30
x = tf.placeholder("float", [None, n_input])
y = tf.placeholder("float", [None])
def multilayer_perceptron(_X, _weights, _biases):
#Hidden layer with RELU activation
layer_1 = tf.nn.relu(tf.add(tf.matmul(_X, _weights['h1']), _biases['b1']))
#Hidden layer with RELU activationn_hidden_3
layer_2 = tf.nn.relu(tf.add(tf.matmul(layer_1, _weights['h2']), _biases['b2']))
layer_3 = tf.nn.relu(tf.add(tf.matmul(layer_2, _weights['h3']), _biases['b3']))
return tf.matmul(layer_3, weights['out']) + biases['out']
weights = {
'h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),
'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
'h3': tf.Variable(tf.random_normal([n_hidden_2, n_hidden_3])),
'out': tf.Variable(tf.random_normal([n_hidden_3, 1]))
}
biases = {
'b1': tf.Variable(tf.random_normal([n_hidden_1])),
'b2': tf.Variable(tf.random_normal([n_hidden_2])),
'b3': tf.Variable(tf.random_normal([n_hidden_3])),
'out': tf.Variable(tf.random_normal([1]))
}
pred = multilayer_perceptron(x, weights, biases)
n_pred = tf.mul(pred, tf.convert_to_tensor(10000.00))
cost = tf.nn.sigmoid_cross_entropy_with_logits(n_pred, y)
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(cost)
init = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init)
# Training cycle
for epoch in range(training_epochs):
avg_cost = 0
total_batch = int(feed_input.train._num_examples / batch_size)
# Loop over all batches
for i in range(total_batch):
batch_xs, batch_ys = feed_input.train.next_batch(batch_size)
# Fit training using batch data
sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys})
# Compute average loss
avg_cost += sess.run(cost, feed_dict={x: batch_xs, y: batch_ys}) / total_batch
# Display logs per epoch step
if epoch % display_step == 0:
print "Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(avg_cost)
print "Optimization Finished!"
runfile('/mnt/sdb6/Projects/StockML/demo1.py',
wdir='/mnt/sdb6/Projects/StockML')
Reloaded modules: tensorflow.python.ops.nn_grad,
tensorflow.python.training.momentum,
. . . .
tensorflow.python.util.protobuf,
google.protobuf.internal.enum_type_wrapper,
tensorflow.python.ops.nn_ops, tensorflow.python,
tensorflow.python.platform.test,
google.protobuf.internal.api_implementation, tensorflow,
google.protobuf.internal.encoder
Traceback (most recent call last):
File "", line 1, in
runfile('/mnt/sdb6/Projects/StockML/demo1.py', wdir='/mnt/sdb6/Projects/StockML')
File
"/usr/lib/python2.7/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py",
line 685, in runfile
execfile(filename, namespace)
File
"/usr/lib/python2.7/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py",
line 78, in execfile
builtins.execfile(filename, *where)
File "/mnt/sdb6/Projects/StockML/demo1.py", line 69, in
sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys})
File
"/home/rammak/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py",
line 345, in run
results = self._do_run(target_list, unique_fetch_targets, feed_dict_string)
File
"/home/rammak/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py",
line 406, in _do_run
except tf_session.StatusNotOK as e:
AttributeError: 'module' object has no attribute 'StatusNotOK'
Protobuf error is usually an installation issue , run it in a virtual env
# On Mac:
$ sudo easy_install pip # If pip is not already installed
$ sudo pip install --upgrade virtualenv
Next, set up a new virtualenv environment. To set it up in the directory ~/tensorflow, run:
$ virtualenv --system-site-packages ~/tensorflow
$ cd ~/tensorflow
Then activate the virtualenv:
$ source bin/activate # If using bash
$ source bin/activate.csh # If using csh
(tensorflow)$ # Your prompt should change
Inside the virtualenv, install TensorFlow:
(tensorflow)$ pip install --upgrade https://storage.googleapis.com/tensorflow/mac/tensorflow-0.5.0-py2-none-any.whl
You can then run your TensorFlow program like:
(tensorflow)$ python tensorflow/models/image/mnist/convolutional.py
# When you are done using TensorFlow:
(tensorflow)$ deactivate # Deactivate the virtualenv
$ # Your prompt should change back
If you just begin to learn TensorFlow, I would suggest you trying out examples in TensorFlow/skflow first and then once you are more familiar with TensorFlow it would be fairly easy for you to insert TensorFlow code to build a custom model you want (there are also examples for this).
Hope those examples for images and text understanding could get you started and let us know if you encounter any issues! (post issues or tag skflow in SO).
Change your logging level from WARN to INFO, so that can get a better visualization of the error you're getting.
For knowledge purpose, you should know there are 5 logging levels:
DEBUG
INFO
WARN
ERROR
FATAL

how to download a package with python-apt?

I want to know as title says: how to download a package using python-apt API? Only download it, don't to install it, just like does the command:
apt-get download ${package_name}
I'm using python v2.7.5-5ubuntu3 with python-apt v0.9.3.5 on Ubuntu 14.04.
You can use this script
#!/usr/bin/python
import apt
import os, sys
if len(sys.argv) != 2:
print 'Usage: apt-download pkg_name'
sys.exit()
cache = apt.Cache()
pkg = cache[sys.argv[1]]
cmd = 'wget ' + pkg.candidate.uri
print cmd
os.system(cmd)
It works well with Python 2.7.8 and python-apt 0.9.3.11 but it should also work with your configuration.