Error deploying on Heroku (ReportLab) - django

I want to deploy a python / django application on Heroku. In the local environment everything works fine. I use pip to install packages. My requirements.txt is the following:
Django==1.6.2
Pillow==2.4.0
dj-database-url==0.3.0
dj-static==0.0.5
django-ckeditor-updated==4.2.8
django-toolbelt==0.0.1
gunicorn==18.0
html5lib==1.0b3
mongoengine==0.8.7
psycopg2==2.5.2
pyPdf==1.13
pymongo==2.7
pystache==0.5.3
reportlab==3.1.8
six==1.6.1
static==1.0.2
wsgiref==0.1.2
xhtml2pdf==0.0.5
But when deploying on Heroku with the command "git push heroku master" I get the error on "ReportLab" installation:
gcc -pthread -shared build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/_renderPM.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/gt1/gt1-parset1.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/gt1/gt1-dict.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/gt1/gt1-namecontext.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/gt1/gt1-region.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_vpath_bpath.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_rgb_pixbuf_affine.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_rgb_svp.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_svp.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_svp_vpath.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_svp_vpath_stroke.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_svp_ops.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_vpath.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_vpath_dash.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_affine.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_rect.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_rgb_affine.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_rgb_affine_private.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_rgb.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_rgb_rgba_affine.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_svp_intersect.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_svp_render_aa.o build/temp.linux-x86_64-2.7/tmp/pip_build_u4591/reportlab/src/rl_addons/renderPM/libart_lgpl/art_misc.o -L/usr/local/lib -L/usr/lib -L/app/.heroku/python/lib -L/app/.heroku/python/lib -lfreetype -lpython2.7 -o build/lib.linux-x86_64-2.7/reportlab/graphics/_renderPM.so
/usr/bin/ld: /usr/local/lib/libpython2.7.a(abstract.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/libpython2.7.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
----------------------------------------
Cleaning up...
Command /app/.heroku/python/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_u4591/reportlab/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-F4RTRK-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_u4591/reportlab
Traceback (most recent call last):
File "/app/.heroku/python/bin/pip", line 9, in <module>
load_entry_point('pip==1.5.6', 'console_scripts', 'pip')()
File "/app/.heroku/python/lib/python2.7/site-packages/pip-1.5.6-py2.7.egg/pip/__init__.py", line 185, in main
return command.main(cmd_args)
File "/app/.heroku/python/lib/python2.7/site-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 161, in main
text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 38: ordinal not in range(128)
! Push rejected, failed to compile Python app
To git#heroku.com:mysterious-oasis-7382.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'git#heroku.com:mysterious-oasis-7382.git'
Jeromes-MacBook-Pro:diagnosystem_proj Jerome$
Your help is appreciated to resolve this issue.

The problem seems to be caused by the presence of a static Python 2.7 library in /usr/local/lib/libpython2.7.a ; to install Reportlab on Heroku correctly using the python-2.7.7 runtime, I changed the order of the directories enumerated by the __call__ function in the inc_lib_dirs class of Reportlab's setup.py file as follows:
Original:
class inc_lib_dirs:
L = None
I = None
def __call__(self):
if self.L is None:
L = []
I = []
if platform == "cygwin":
aDir(L, os.path.join("/usr/lib", "python%s" % sys.version[:3], "config"))
elif platform == "darwin":
# attempt to make sure we pick freetype2 over other versions
aDir(I, "/sw/include/freetype2")
aDir(I, "/sw/lib/freetype2/include")
# fink installation directories
aDir(L, "/sw/lib")
aDir(I, "/sw/include")
# darwin ports installation directories
aDir(L, "/opt/local/lib")
aDir(I, "/opt/local/include")
aDir(I, "/usr/local/include")
aDir(L, "/usr/local/lib")
aDir(I, "/usr/include")
aDir(L, "/usr/lib")
aDir(I, "/usr/include/freetype2")
prefix = sysconfig.get_config_var("prefix")
if prefix:
aDir(L, pjoin(prefix, "lib"))
aDir(I, pjoin(prefix, "include"))
self.L=L
self.I=I
return self.I,self.L
inc_lib_dirs=inc_lib_dirs()
Edited:
class inc_lib_dirs:
L = None
I = None
def __call__(self):
if self.L is None:
L = []
I = []
if platform == "cygwin":
aDir(L, os.path.join("/usr/lib", "python%s" % sys.version[:3], "config"))
elif platform == "darwin":
# attempt to make sure we pick freetype2 over other versions
aDir(I, "/sw/include/freetype2")
aDir(I, "/sw/lib/freetype2/include")
# fink installation directories
aDir(L, "/sw/lib")
aDir(I, "/sw/include")
# darwin ports installation directories
aDir(L, "/opt/local/lib")
aDir(I, "/opt/local/include")
prefix = sysconfig.get_config_var("prefix")
if prefix:
aDir(L, pjoin(prefix, "lib"))
aDir(I, pjoin(prefix, "include"))
aDir(I, "/usr/local/include")
aDir(L, "/usr/local/lib")
aDir(I, "/usr/include")
aDir(L, "/usr/lib")
aDir(I, "/usr/include/freetype2")
self.L=L
self.I=I
return self.I,self.L
inc_lib_dirs=inc_lib_dirs()
I'd love to find a cleaner solution to avoid keeping a modified version in my egg repository.

Related

How to run moderngl in Colab?

I'm trying to run moderngl in Colab. I installed it and ran a virtual display:
!sudo apt-get update --fix-missing && apt-get -qqq install x11-utils > /dev/null
!sudo apt-get update --fix-missing && apt-get -qqq install xvfb > /dev/null
!python3 -m pip install -U -qqq moderngl
!python3 -m pip install -U -qqq moderngl-window
!python3 -m pip install -U -qqq pyvirtualdisplay
from pyvirtualdisplay import Display
display = Display(visible=0, size=(960, 540)).start()
import moderngl
ctx = moderngl.create_standalone_context()
buf = ctx.buffer(b'Hello World!') # allocated on the GPU
buf.read()
b'Hello World!'
It printed as expected, but when I run an example I see the error:
!python3 /content/moderngl/examples/basic_alpha_blending.py --window pyglet
2020-03-28 10:25:48,312 - moderngl_window - INFO - Attempting to load window class: moderngl_window.context.pyglet.Window
Traceback (most recent call last):
File "/content/moderngl/examples/basic_alpha_blending.py", line 74, in <module>
AlphaBlending.run()
File "/content/moderngl/examples/ported/_example.py", line 21, in run
mglw.run_window_config(cls)
File "/usr/local/lib/python3.6/dist-packages/moderngl_window/__init__.py", line 185, in run_window_config
cursor=show_cursor if show_cursor is not None else True,
File "/usr/local/lib/python3.6/dist-packages/moderngl_window/context/pyglet/window.py", line 54, in __init__
config=config,
File "/usr/local/lib/python3.6/dist-packages/pyglet/window/xlib/__init__.py", line 165, in __init__
super(XlibWindow, self).__init__(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pyglet/window/__init__.py", line 588, in __init__
config = screen.get_best_config(config)
File "/usr/local/lib/python3.6/dist-packages/pyglet/canvas/base.py", line 194, in get_best_config
raise window.NoSuchConfigException()
pyglet.window.NoSuchConfigException
I also tried with another virtual display, but the result is the same:
!python3 -m pip install -U -qqq xvfbwrapper
from xvfbwrapper import Xvfb
display = Xvfb(width=960, height=540).start()
pyglet.window.NoSuchConfigException
In Google Colab you can use the EGL backend with moderngl 5.6.
ctx = moderngl.create_context(standalone=True, backend='egl')
print(ctx.info)
Output (partial):
{
'GL_VENDOR': 'NVIDIA Corporation',
'GL_RENDERER': 'Tesla P100-PCIE-16GB/PCIe/SSE2',
'GL_VERSION': '3.3.0 NVIDIA 418.67',
....
}
moderngl creates an OpenGL 3.3 core context by default. If you need a higher context version you can pass in require=430 for OpenGL 4.3 for example. I don't know what these Tesla cards support.
There is a standard example in moderngl for this. It would be able to create the standard RGB triangle: https://github.com/moderngl/moderngl/blob/master/examples/headless_egl.py
The underlying library creating the contexts is glcontext (https://github.com/moderngl/glcontext).
if you are using the moderngl-window package you have to use the headless.Window because pyglet currently is not able to work in headless mode (It might in the future: https://github.com/pyglet/pyglet/issues/51)
If you run into issue make an issue in the moderngl project: https://github.com/moderngl/moderngl or invade their discord server.

`nix-build` fails because Python wants something that's unavailable, without saying what it wants

(At least I think that's the problem.)
The error
When I run nix-build -A serialosc from the root of nixpkgs, nix prints the following:
[jeff#jbb-dell:~/nix/nixpkgs-fork]$ nix-build -A serialosc
these derivations will be built:
/nix/store/p40wvn28grb8nrcd6scbxhhjqw4495kk-serialosc.drv
building '/nix/store/p40wvn28grb8nrcd6scbxhhjqw4495kk-serialosc.drv'...
unpacking sources
unpacking source archive /nix/store/5zj802wfjd0ima92lpzzsqdjqvrnrwf9-serialosc
source root is serialosc
patching sources
configuring
configure flags: --prefix=/nix/store/b6qzvzmwjdi29nnzc769904wc0mj1cds-serialosc configure
Setting top to : /build/serialosc
Setting out to : /build/serialosc/build
Checking for 'gcc' (C compiler) : gcc
Checking for working poll() : yes
Checking for libudev : yes
Checking for libmonome : yes
Checking for liblo : yes
Checking for function strdup : yes
Checking for function _strdup : not found
Checking for function strndup : yes
Checking for function strcasecmp : yes
Checking for header unistd.h : yes
Checking for header dns_sd.h : yes
Checking for library dl : yes
Traceback (most recent call last):
File "/build/serialosc/.waf-1.8.5-3fc7b0f5070c04bfd3f2489448471a84/waflib/Scripting.py", line 103, in waf_entry_point
run_commands()
File "/build/serialosc/.waf-1.8.5-3fc7b0f5070c04bfd3f2489448471a84/waflib/Scripting.py", line 164, in run_commands
ctx=run_command(cmd_name)
File "/build/serialosc/.waf-1.8.5-3fc7b0f5070c04bfd3f2489448471a84/waflib/Scripting.py", line 155, in run_command
ctx.execute()
File "/build/serialosc/.waf-1.8.5-3fc7b0f5070c04bfd3f2489448471a84/waflib/Configure.py", line 92, in execute
super(ConfigurationContext,self).execute()
File "/build/serialosc/.waf-1.8.5-3fc7b0f5070c04bfd3f2489448471a84/waflib/Context.py", line 92, in execute
self.recurse([os.path.dirname(g_module.root_path)])
File "/build/serialosc/.waf-1.8.5-3fc7b0f5070c04bfd3f2489448471a84/waflib/Context.py", line 133, in recurse
user_function(self)
File "/build/serialosc/wscript", line 259, in configure
stderr=devnull).decode().strip()
File "/nix/store/lbrpma3528hq7gwn8ffbnwaxvsqah0bb-python-2.7.17/lib/python2.7/subprocess.py", line 216, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/nix/store/lbrpma3528hq7gwn8ffbnwaxvsqah0bb-python-2.7.17/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/nix/store/lbrpma3528hq7gwn8ffbnwaxvsqah0bb-python-2.7.17/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
builder for '/nix/store/p40wvn28grb8nrcd6scbxhhjqw4495kk-serialosc.drv' failed with exit code 2
error: build of '/nix/store/p40wvn28grb8nrcd6scbxhhjqw4495kk-serialosc.drv' failed
[jeff#jbb-dell:~/nix/nixpkgs-fork]$
My suspicion
The problem appears to arise in lib/python2.7/subprocess.py. I found a version of that file online, and the definition of _execute_child seems to make reference to some absolute paths -- for instance, in this passage:
if shell:
args = ["/bin/sh", "-c"] + args
if executable:
args[0] = executable
Therefore my guess is that Nix is building in a sandbox that doesn't offer whatever Python expects. But the error doesn't tell me what Python was hoping to find.
The context
(You might not need this information.)
I'm trying to build serialosc on NixOS 19.09. Here's my fork of nixpkgs, including my broken recipe for serialosc. The fork also includes my libmonome package, which works, and which serialosc depends on, and which is not yet part of the official nixpkgs repo. I've given hard links to the specific commit; they shouldn't change when I update the fork.
Cross-posting from
https://discourse.nixos.org/t/nix-build-fails-because-python-wants-something-thats-unavailable-without-saying-what-it-wants/5675/2
so that it could help people who only find this, answer might continue on NixOS discourse. I tried to keep it helpful for general NixOS writing a derivation debugging, with some Python bits
fetchgit
First, let's help with the fetchgit part in your recipe
src = fetchgit {
# Once, it seemed to finish, and I don't know how! Then it crashed with:
# Switched to a new branch 'fetchgit'
# removing `.git'...
# hash mismatch in fixed-output derivation '/nix/store/5zj802wfjd0ima92lpzzsqdjqvrnrwf9-serialosc':
# wanted: sha256:1vqcxi32wc4pklbddllflkaigkfvd4ykwrjqccayvrk10dx1sna3
# got: sha256:1zmzjasv21ix7i7s58a31k0025ji32hv2jm2ww6s0xhjmr5ax34j
# This way it [[gets pretty far]].
# The fetchSubmodules value I set again seems to have no effect.
url = https://github.com/monome/serialosc.git;
rev = "v1.4.1";
sha256 = "1zmzjasv21ix7i7s58a31k0025ji32hv2jm2ww6s0xhjmr5ax34j";
Perhaps there was a force push? Not sure why it would stop working with the hash
I found that the fetchSubmodules has no effect if you have the hash of the one without the submodules, use nix-prefetch-git --fetch-submodules https://github.com/monome/serialosc which gives
{
"url": "https://github.com/monome/serialosc",
"rev": "cec0ea76b2d5f69afa74d3ffc14a0950e32a7914",
"date": "2019-06-09T21:46:13+02:00",
"sha256": "03qkzslhih72idwafgfxmkwp5v3x048njh0c682phw2ks11plmbp",
"fetchSubmodules": true
}
so use
fetchgit {
url = https://github.com/monome/serialosc;
rev = "cec0ea76b2d5f69afa74d3ffc14a0950e32a7914";
# or rev = version; but that might break with force pushes
sha256 = "03qkzslhih72idwafgfxmkwp5v3x048njh0c682phw2ks11plmbp";
fetchSubmodules = true;
}
wafHook
There is a (slight) wafHook documentation in https://nixos.org/nixpkgs/manual/
Debugging error
It seems to want to do git stuff, here is how I debugged it:
$ nix-shell https://github.com/JeffreyBenjaminBrown/nixpkgs/archive/2bbd131811991bec9ae6b296b5cd1441640c306c.tar.gz -A serialosc --pure
nix-shell $ mkdir /tmp/delme-serialosc
nix-shell $ cd !$
# To find out what a build would do
nix-shell $ type genericBuild
...
# Doing the `if [ -z "${phases:-}" ]; then ...` bit gives us `echo $phases`
# unpackPhase patchPhase configurePhase buildPhase checkPhase installPhase fixupPhase installCheckPhase distPhase
nix-shell $ ${unpackPhase:-unpackPhase}
nix-shell $ cd $sourceRoot
nix-shell $ ${patchPhase:-patchPhase}
nix-shell $ ${configurePhase:-configurePhase}
# Error happens here
nix-shell $ type ${configurePhase:-configurePhase}
# Do the steps until you find it stops in `python ...`
# Python debug info: https://docs.python.org/3/library/pdb.html
nix-shell $ python -m pdb "$wafPath" "${flagsArray[#]}";
(Pdb) continue # or cont, or just c
...
(Pdb) break /nix/store/lbrpma3528hq7gwn8ffbnwaxvsqah0bb-python-2.7.17/lib/python2.7/subprocess.py:1047
(Pdb) continue
> /nix/store/lbrpma3528hq7gwn8ffbnwaxvsqah0bb-python-2.7.17/lib/python2.7/subprocess.py(1047)_execute_child()
-> raise child_exception
(Pdb) up
> /nix/store/lbrpma3528hq7gwn8ffbnwaxvsqah0bb-python-2.7.17/lib/python2.7/subprocess.py(394)__init__()
-> errread, errwrite)
(Pdb) # empty is equivalent to do last command, i.e. up in this case
> /nix/store/lbrpma3528hq7gwn8ffbnwaxvsqah0bb-python-2.7.17/lib/python2.7/subprocess.py(216)check_output()
-> process = Popen(stdout=PIPE, *popenargs, **kwargs)
(Pdb) print(popenargs)
(['git', 'rev-parse', '--verify', '--short', 'HEAD'],)
But you will find you might need to figure a way to avoid waf using git as it cannot access the network unless it's a fixed-output derivation (e.g. those fetching sources). Hopefully the bit about fetchSubmodules will fix it!
Edit: I didn’t read past that it tries to use git, it tries to figure out what version it is, you should somehow make it avoid using git to do that, e.g. grep -r 'rev-parse' suggests editing the wscript file
Edit 2: E.g. something like
--- wscript-old 2020-01-29 15:55:54.100976818 +0000
+++ wscript 2020-01-29 15:55:58.128977024 +0000
## -257,7 +257,7 ##
conf.env.GIT_COMMIT = subprocess.check_output(
["git", "rev-parse", "--verify", "--short", "HEAD"],
stderr=devnull).decode().strip()
- except subprocess.CalledProcessError:
+ except (subprocess.CalledProcessError, OSError):
conf.env.GIT_COMMIT = ''
conf.define("VERSION", VERSION)
perhaps it could be upstreamed too (this leaves the git commit ID as empty in this case)? Or just use substituteInPlace or something to put in the actual commit ID to be
--- wscript-old 2020-01-29 15:55:54.100976818 +0000
+++ wscript 2020-01-29 15:57:56.177983054 +0000
## -249,16 +249,7 ##
conf.env.VERSION = VERSION
- try:
- import os
-
- devnull = open(os.devnull, 'w')
-
- conf.env.GIT_COMMIT = subprocess.check_output(
- ["git", "rev-parse", "--verify", "--short", "HEAD"],
- stderr=devnull).decode().strip()
- except subprocess.CalledProcessError:
- conf.env.GIT_COMMIT = ''
+ conf.env.GIT_COMMIT = 'cec0ea76b2d5f69afa74d3ffc14a0950e32a7914'
conf.define("VERSION", VERSION)
conf.define("_GNU_SOURCE", 1)

Building Gstreamer Editing Services fails

I run ./autogen.sh inside the cloned repo and it fails saying the following:
configure: No package 'gstreamer-plugins-base-1.0' found
configure: error: no gstreamer-plugins-base-1.0 >= 1.14.1 (GStreamer Base Plugins) found
configure failed
I have gstreamer (base, good, bad and ugly) installed on my Ubuntu. The package name that the build script looks for is gstreamer-plugins-base-1.0 where as the system package is by the name gstreamer1.0-plugins-base.
Digging into the autoconf setup I found the following:
if test -z $GSTPB_PLUGINS_DIR; then
GSTPB_PLUGINS_DIR=`$PKG_CONFIG --variable=pluginsdir gstreamer-plugins-base-[$1]`
if test -z $GSTPB_PLUGINS_DIR; then
AC_MSG_ERROR(
[no pluginsdir set in GStreamer Base Plugins pkg-config file])
fi
fi
Shouldn't it be gstreamer[$1]-plugins-base? Am I missing something here?
Update:
Fixed the above by installing libgstreamer1.0-dev and libgstreamer-plugins-base1.0-dev dev packages
sudo apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev
In case GIntrospection isn't installed by default, run the following
sudo apt-get build-dep gstreamer1.0
./autogen.sh would complete and make && sudo make install will run fine too.
Current status: Examples don't build stable binaries. Running c examples segfault and python example, simple.py, complains of missing GES in the namespace.
Traceback (most recent call last):
File "simple.py", line 26, in <module>
gi.require_version('GES', '1.0')
File "/usr/lib/python2.7/dist-packages/gi/__init__.py", line 130, in require_version
raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace GES not available
Just for reference: Simple.py looks like this
import gi
gi.require_version('Gst', '1.0')
gi.require_version('GES', '1.0')
from gi.repository import Gst, GES, GLib # noqa
class Simple:
def __init__(self, uri):
timeline = GES.Timeline.new_audio_video()
self.project = timeline.get_asset()
self.project.connect("asset-added", self._asset_added_cb)
self.project.connect("error-loading-asset", self._error_loading_asset_cb)
self.project.create_asset(uri, GES.UriClip)
self.layer = timeline.append_layer()
self._create_pipeline(timeline)
self.loop = GLib.MainLoop()
def _create_pipeline(self, timeline):
self.pipeline = GES.Pipeline()
self.pipeline.set_timeline(timeline)
bus = self.pipeline.get_bus()
bus.add_signal_watch()
bus.connect("message", self.bus_message_cb)
def bus_message_cb(self, unused_bus, message):
if message.type == Gst.MessageType.EOS:
print("eos")
self.loop.quit()
elif message.type == Gst.MessageType.ERROR:
error = message.parse_error()
print("error %s" % error[1])
self.loop.quit()
def start(self):
self.loop.run()
def _asset_added_cb(self, project, asset):
self.layer.add_asset(asset, 0, 0, Gst.SECOND * 5, GES.TrackType.UNKNOWN)
self.pipeline.set_state(Gst.State.PLAYING)
def _error_loading_asset_cb(self, project, error, asset_id, type):
print("Could not load asset %s: %s" % (asset_id, error))
self.loop.quit()
if __name__ == "__main__":
if len(os.sys.argv) != 2:
print("You must specify a file URI")
exit(-1)
Gst.init(None)
GES.init()
simple = Simple(os.sys.argv[1])
simple.start()
Running the C example, simple1.c, fails with the following:
(simple1:15606): GLib-GObject-WARNING **: 12:42:28.910: invalid (NULL) pointer instance
(simple1:15606): GLib-GObject-CRITICAL **: 12:42:28.910: g_signal_connect_data: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed
(simple1:15606): GLib-GObject-WARNING **: 12:42:28.910: invalid (NULL) pointer instance
(simple1:15606): GLib-GObject-CRITICAL **: 12:42:28.910: g_signal_connect_data: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed
(simple1:15606): GLib-GObject-CRITICAL **: 12:42:28.910: g_object_set: assertion 'G_IS_OBJECT (object)' failed
[1] 15606 segmentation fault (core dumped) ./simple1 ~/Downloads/out.mp4
Running gdb,
gst-editing-services/examples/c/simple1": not in executable format: File format not recognized
Update
Rebuilt the examples using Meson build system. This enabled running the bins in gdb. Got the following
Program received signal SIGSEGV, Segmentation fault.
ges_track_constructed (object=<optimized out>) at ../ges/ges-track.c:506
506 componame =
Indicating that it failed at ges-track.c. Relevant code below:
if (self->type == GES_TRACK_TYPE_VIDEO) {
componame =
g_strdup_printf ("video_%s", GST_OBJECT_NAME (self->priv->composition));
} else if (self->type == GES_TRACK_TYPE_AUDIO) {
componame = // This is where it errirs
g_strdup_printf ("audio_%s", GST_OBJECT_NAME (self->priv->composition));
}
Stepping into it line by line. The following was revealed.
0x00007ffff701c2cd in __GI__dl_catch_exception (exception=exception#entry=0x7fffffffc980,
operate=0x7ffff54530d0 <dlsym_doit>, args=0x7fffffffc9f0) at dl-error-skeleton.c:194
194 dl-error-skeleton.c: No such file or directory.
It looks like it has to do with GES internals. Internally, glib objects are not created right and are all NULL. Why is why you have the invalid (NULL) pointer instance.
I was able to run the examples on my mac. Package mismatches seems to be the issue here.

Error in TensorFlow program

I am learning TensorFlow and I stumble upon this example code for creating simple multi-layer sigmoid network. The program in the link is for MNIST database and hand written digit classification.
I want to train a network for regression task. I have 30 inputs(float) which is used to predict one output(float). So I tweaked the code to change the task from classification to regression.
My problem is that I'm getting an error in tf.Session.run(). The code and the error log is given below.
import test2
import tensorflow as tf
feed_input = test2.read_data_sets()
learning_rate = 0.001
training_epochs = 100
batch_size = 1716
display_step = 1
n_hidden_1 = 256
n_hidden_2 = 256
n_hidden_3 = 256
n_input = 30
x = tf.placeholder("float", [None, n_input])
y = tf.placeholder("float", [None])
def multilayer_perceptron(_X, _weights, _biases):
#Hidden layer with RELU activation
layer_1 = tf.nn.relu(tf.add(tf.matmul(_X, _weights['h1']), _biases['b1']))
#Hidden layer with RELU activationn_hidden_3
layer_2 = tf.nn.relu(tf.add(tf.matmul(layer_1, _weights['h2']), _biases['b2']))
layer_3 = tf.nn.relu(tf.add(tf.matmul(layer_2, _weights['h3']), _biases['b3']))
return tf.matmul(layer_3, weights['out']) + biases['out']
weights = {
'h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),
'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
'h3': tf.Variable(tf.random_normal([n_hidden_2, n_hidden_3])),
'out': tf.Variable(tf.random_normal([n_hidden_3, 1]))
}
biases = {
'b1': tf.Variable(tf.random_normal([n_hidden_1])),
'b2': tf.Variable(tf.random_normal([n_hidden_2])),
'b3': tf.Variable(tf.random_normal([n_hidden_3])),
'out': tf.Variable(tf.random_normal([1]))
}
pred = multilayer_perceptron(x, weights, biases)
n_pred = tf.mul(pred, tf.convert_to_tensor(10000.00))
cost = tf.nn.sigmoid_cross_entropy_with_logits(n_pred, y)
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(cost)
init = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init)
# Training cycle
for epoch in range(training_epochs):
avg_cost = 0
total_batch = int(feed_input.train._num_examples / batch_size)
# Loop over all batches
for i in range(total_batch):
batch_xs, batch_ys = feed_input.train.next_batch(batch_size)
# Fit training using batch data
sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys})
# Compute average loss
avg_cost += sess.run(cost, feed_dict={x: batch_xs, y: batch_ys}) / total_batch
# Display logs per epoch step
if epoch % display_step == 0:
print "Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(avg_cost)
print "Optimization Finished!"
runfile('/mnt/sdb6/Projects/StockML/demo1.py',
wdir='/mnt/sdb6/Projects/StockML')
Reloaded modules: tensorflow.python.ops.nn_grad,
tensorflow.python.training.momentum,
. . . .
tensorflow.python.util.protobuf,
google.protobuf.internal.enum_type_wrapper,
tensorflow.python.ops.nn_ops, tensorflow.python,
tensorflow.python.platform.test,
google.protobuf.internal.api_implementation, tensorflow,
google.protobuf.internal.encoder
Traceback (most recent call last):
File "", line 1, in
runfile('/mnt/sdb6/Projects/StockML/demo1.py', wdir='/mnt/sdb6/Projects/StockML')
File
"/usr/lib/python2.7/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py",
line 685, in runfile
execfile(filename, namespace)
File
"/usr/lib/python2.7/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py",
line 78, in execfile
builtins.execfile(filename, *where)
File "/mnt/sdb6/Projects/StockML/demo1.py", line 69, in
sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys})
File
"/home/rammak/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py",
line 345, in run
results = self._do_run(target_list, unique_fetch_targets, feed_dict_string)
File
"/home/rammak/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py",
line 406, in _do_run
except tf_session.StatusNotOK as e:
AttributeError: 'module' object has no attribute 'StatusNotOK'
Protobuf error is usually an installation issue , run it in a virtual env
# On Mac:
$ sudo easy_install pip # If pip is not already installed
$ sudo pip install --upgrade virtualenv
Next, set up a new virtualenv environment. To set it up in the directory ~/tensorflow, run:
$ virtualenv --system-site-packages ~/tensorflow
$ cd ~/tensorflow
Then activate the virtualenv:
$ source bin/activate # If using bash
$ source bin/activate.csh # If using csh
(tensorflow)$ # Your prompt should change
Inside the virtualenv, install TensorFlow:
(tensorflow)$ pip install --upgrade https://storage.googleapis.com/tensorflow/mac/tensorflow-0.5.0-py2-none-any.whl
You can then run your TensorFlow program like:
(tensorflow)$ python tensorflow/models/image/mnist/convolutional.py
# When you are done using TensorFlow:
(tensorflow)$ deactivate # Deactivate the virtualenv
$ # Your prompt should change back
If you just begin to learn TensorFlow, I would suggest you trying out examples in TensorFlow/skflow first and then once you are more familiar with TensorFlow it would be fairly easy for you to insert TensorFlow code to build a custom model you want (there are also examples for this).
Hope those examples for images and text understanding could get you started and let us know if you encounter any issues! (post issues or tag skflow in SO).
Change your logging level from WARN to INFO, so that can get a better visualization of the error you're getting.
For knowledge purpose, you should know there are 5 logging levels:
DEBUG
INFO
WARN
ERROR
FATAL

How to set a compiler for codepy.toolchain?

These two lines were taken from the ThrustInterop.py found in PyCuda/Examples
import codepy.toolchain
codepy.toolchain.guess_toolchain()
I understand that this should return the Toolchain instance for a C++ compiler found in $HOME/.aksetup-defaults.py.
BOOST_COMPILER = 'gcc'
BOOST_INC_DIR = ['/usr/local/Cellar/boost/1.57.0/include/']
BOOST_LIB_DIR = ['/usr/local/Cellar/boost/1.57.0/lib/']
BOOST_PYTHON_LIBNAME = ['boost_python']
BOOST_THREAD_LIBNAME = ['boost_thread']
CUDADRV_LIBNAME = ['cuda']
CUDADRV_LIB_DIR = ['${CUDA_ROOT}/lib']
CUDART_LIBNAME = ['cudart']
CUDART_LIB_DIR = ['${CUDA_ROOT}/lib']
CUDA_ENABLE_CURAND = True
CUDA_ENABLE_GL = True
CUDA_INC_DIR = ['${CUDA_ROOT}/include']
CUDA_TRACE = False
CURAND_LIBNAME = ['curand']
CURAND_LIB_DIR = ['${CUDA_ROOT}/lib']
USE_SHIPPED_BOOST = False
Under Python 2.7.6, I get the following exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Python/2.7/site-packages/codepy/toolchain.py", line 433, in guess_toolchain
raise ToolchainGuessError("unknown compiler")
codepy.toolchain.ToolchainGuessError: unknown compiler
On a Cluster with Python 3.4.1:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/cluster/home03/math/echeverl/local2/lib/python3.4/site-packages/codepy/toolchain.py", line 439, in guess_nvcc_toolchain
gcc_kwargs = _guess_toolchain_kwargs_from_python_config()
File "/cluster/home03/math/echeverl/local2/lib/python3.4/site-packages/codepy/toolchain.py", line 398, in _guess_toolchain_kwargs_from_python_config
so_ext=make_vars["SO"],
KeyError: 'SO'
After looking at the library I found that it doesn't recognize clang so I modified 'guess_toolchain' in toolchain.py
def guess_toolchain():
"""Guess and return a :class:`Toolchain` instance.
Raise :exc:`ToolchainGuessError` if no toolchain could be found.
"""
kwargs = _guess_toolchain_kwargs_from_python_config()
from pytools.prefork import call_capture_output
result, version, stderr = call_capture_output([kwargs["cc"], "--version"])
if result != 0:
raise ToolchainGuessError("compiler version query failed: "+stderr)
#Original Code
# if "Free Sofware Foundation" in version:
#Modified line to recognize clang
if ("Free Software Foundation" in version) or ("clang" in version):
if "-Wstrict-prototypes" in kwargs["cflags"]:
kwargs["cflags"].remove("-Wstrict-prototypes")
if "darwin" in version:
# Are we running in 32-bit mode?
# The python interpreter may have been compiled as a Fat binary
# So we need to check explicitly how we're running
# And update the cflags accordingly
import sys
if sys.maxint == 0x7fffffff:
kwargs["cflags"].extend(['-arch', 'i386'])
return GCCToolchain(**kwargs)
else:
raise ToolchainGuessError("unknown compiler")
How can I tell codepy.toolchain which compiler to use?
Setting the CC environment variable to an appropriate value might help. Also check your Python Makefile, stored under config in your Python lib directory. That's where the config information comes from in the first place.
I modified the file ~/.aksetup-defaults.py to:
#For CUDA
CUDA_ROOT=['/Developer/NVIDIA/CUDA-5.0/']
CUDADRV_LIB_DIR=['/Developer/NVIDIA/CUDA-5.0/lib']
CUDA_INC_DIR=['/Developer/NVIDIA/CUDA-5.0/include']
#For Boost
BOOST_INC_DIR=['${HOME}/pool/include/']
BOOST_LIB_DIR=['${HOME}/pool/lib']
USE_SHIPPED_BOOST=False
This solved the issue with toolchain, but I'm still trying to figure out how to compile.