I trying to import module by __import__ like this:
>>> mod = __import__('x.y.z')
But I only got x:
>>> print mod
>>> <module 'x' from '...'>
How should I do to import z ? I tried like this, it works but i don't know why.
>>> mod = __import__('x.y.z', {}, {}, [''])
>>> print mod
>>> <module 'x.y.z' from '...'>
I'm really confused about this, and I also have no idea with the globals and locals parameters.
Thx a lot!
Relevant notes from the docs (__import__):
When the name variable is of the form package.module, normally, the top-level package (the name up till the first dot) is returned, not the module named by name. However, when a non-empty fromlist argument is given, the module named by name is returned.
Hence, it's similar to writing import x.y.z which also makes x available (as well as x.y and x.y.z).
Use the importlib module instead. Of which the bare bones are made available in 2.7.
import importlib
z = importlib.import_module("z", "x.y")
# equivalent to
from x.y import z
Related
I've read some answers on this question here and here, however I'm still a bit puzzled by tf.Variable being and/or not being a tf.Tensor.
The linked answers deal with a mutability of tf.Variable and mentioning that tf.Variables maintains their states (when instantiated with default parameter trainable=True).
What makes me still a bit confused is a test case I came across when writing simple unit tests using tf.test.TestCase
Consider the following code snippet. We have a simple class called Foo which has only one property, a tf.Variable initialized to w:
import tensorflow as tf
import numpy as np
class Foo:
def __init__(self, w):
self.w = tf.Variable(w)
Now, let's say you want to test that the instance of Foo has w initialized with tensor of the same dimension as passed in via w. The simplest test case could be written as follows:
import tensorflow as tf
import numpy as np
from foo import Foo
class TestFoo(tf.test.TestCase):
def test_init(self):
w = np.random.rand(3,2)
foo = Foo(w)
init = tf.global_variables_initializer()
with self.test_session() as sess:
sess.run(init)
self.assertShapeEqual(w, foo.w)
if __name__ == '__main__':
tf.test.main()
Now when you run the test you'll get the following error:
======================================================================
ERROR: test_init (__main__.TestFoo)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_foo.py", line 12, in test_init
self.assertShapeEqual(w, foo.w)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/test_util.py", line 1100, in assertShapeEqual
raise TypeError("tf_tensor must be a Tensor")
TypeError: tf_tensor must be a Tensor
----------------------------------------------------------------------
Ran 2 tests in 0.027s
FAILED (errors=1)
You can "get around" this unit test error by doing something like this (i.e. note assertShapeEqual was replaced with assertEqual):
self.assertEqual(list(w.shape), foo.w.get_shape().as_list())
What I'm interested in, though, is the tf.Variable vs tf.Tensor relationship.
What the test error seems to be suggesting is that foo.w is NOT a tf.Tensor, meaning you probably can't use tf.Tensor API on it. Consider, however, the following interactive python session:
$ python3
Python 3.6.3 (default, Oct 4 2017, 06:09:15)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> import numpy as np
>>> w = np.random.rand(3,2)
>>> var = tf.Variable(w)
>>> var.get_shape().as_list()
[3, 2]
>>> list(w.shape)
[3, 2]
>>>
In the session above, we create a variable and run the get_shape() method on it to retrieve its shape dimensions. Now, get_shape() method is a tf.Tensor API method as you can see here.
So to get back to my question, what parts of tf.Tensor API does tf.Variable implement. If the answer is ALL of them, why does the above test case fail?
self.assertShapeEqual(w, foo.w)
with
raise TypeError("tf_tensor must be a Tensor")
I'm pretty sure I'm missing something fundamental here or maybe it's a bug in assertShapeEqual ? I would appreciate if someone could shed some light on this.
I'm using following version of tensorflow on macOS with python3:
tensorflow (1.4.1)
That testing utility function is checking whether a variable implements tf.Tensor
>>> import tensorflow as tf
>>> v = tf.Variable('v')
>>> v
<tf.Variable 'Variable:0' shape=() dtype=string_ref>
>>> isinstance(v, tf.Tensor)
False
The answer appears to be 'no'.
Update:
According to the documentation that is correct:
https://www.tensorflow.org/programmers_guide/variables
Unlike tf.Tensor objects, a tf.Variable exists outside the context of
a single session.run call.
Although:
A tf.Variable represents a tensor whose value can be changed by
running ops on it.
(Not quite sure what 'represents a tensor' means - sounds like a design 'feature')
I saw a post from a few days ago by someone else: pymc3 likelihood math with non-theano function. Even though I think the problem at its core is the same, I thought I would ask with a simpler example:
Inside logp_wrap, I put some made up definition of a likelihood function. It depends on the rv and an observation. In this case I could do this with theano operations, but let's say that I want this function to be more complex and so I cannot use theano.
The problem comes when I try to define the likelihood both in terms of an RV and observations. From what I have seen, this format would work if I was specifying everything in 'logp_wrap' as theano operations.
I have searched around for a solution to this, but haven't found anything where this problem is fully addressed.
The problem in my attempt to do this is actually that the logp_ function is correctly decorated, but the logp_wrap function is only correctly decorated for its input, and not for its output, so I get the error
TypeError: 'TensorVariable' object is not callable.
Would be great if someone had a solution - don't think I am the only one with this problem.
The theano version of this that works (and uses the same function within a function definition) without the #as_op code is here: https://pymc-devs.github.io/pymc3/notebooks/lda-advi-aevb.html?highlight=densitydist (Specifically the sections: "Log-likelihood of documents for LDA" and "LDA model section")
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
"""
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import pymc3 as pm
from theano import as_op
import theano.tensor as T
from scipy.stats import norm
#Some data that we observed
g_observed = [0.0, 1.0, 2.0, 3.0]
#Define a function to calculate the logp without using theano
#This as_op is where the problem is - the input is an rv but the output is a
#function.
#as_op(itypes=[T.dscalar],otypes=[T.dscalar])
def logp_wrap(rv):
#We are not using theano so we wrap the function.
#as_op(itypes=[T.dvector],otypes=[T.dscalar])
def logp_(ob):
#Some made up likelihood -
#The key here is that lp depends on the rv input and the observations
lp = np.log(norm.pdf(rv + ob))
return lp
return logp_
hb1_model = pm.Model()
with hb1_model:
I_mean = pm.Normal('I_mean', mu=0.1, sd=0.05)
xs = pm.DensityDist('x', logp_wrap(I_mean),observed = g_observed)
with hb1_model:
step = pm.Metropolis()
trace = pm.sample(1000, step)
I'm learning to use colorama in Python, so I installed it and I'm able to import the module with no problems from the Primary Prompt.
>>> import colorama
>>> from colorama import *
>>> print(Fore.BLUE + 'BLUE TEXT')
BLUE TEXT
Now, if I create a small piece of code like this:
#!/usr/bin/env python2.7
from colorama import *
print(Fore.BLUE + 'BLUE TEXT')
I get the following message:
File "colorama_Test.py", line 3, in <module>
from colorama import *
File "/home/olg32/Python/colorama_Test.py", line 5, in <module>
print(Fore.BLUE + 'BLUE TEXT')
NameError: name 'Fore' is not defined
Which tells me that the module is not being found. But as mentioned it was installed and tested successfully from the Primary Prompt. Could it be a path definition issue or something like that? This is the current directory where the module is installed:
usr/local/lib/python2.7/dist-packages/colorama-0.3.7-py2.7.egg
Does this path needs to be defined somewhere? Sorry I'm new on Python.
Any help would be appreciated.
Thank you.
Hopefully you have worked out the answer by now but have you tried specifying Fore?
When I use the colorama module I start with this:
import os, colorama
from colorama import Fore,Style,Back #specifying all 3 types
os.system("mode con: cols=120 lines=30") #sometimes colorama doesnt work
#when double clicking a python app so I use this to "prompt" command line
#and then it works fine colorama.init() should work too
Example code:
import os, colorama
from colorama import Fore,Style,Back
os.system("mode con: cols=120 lines=30")
print(Fore.RED + 'some red text')
print(Back.GREEN + 'and with a green background')
print(Style.DIM + 'and in dim text')
print(Style.RESET_ALL)
print('back to normal now')
If this doesnt work for you let me know :)
I have been experimenting with python by creating some programs .The thing is, I have no idea how to import something OUT of the default python directory.
OK
So I did some heavy research and the conclusion is
if u want to access a file saved at different location
use
f = open('E:/somedir/somefile.txt', 'r')
r = f.read()
NOTE: Dont use '\' that were I went wrong.Our system addresses uses '\' So be careful
If you need to just read in a file and not import a module the documentation covers this extensively.
https://docs.python.org/2/tutorial/inputoutput.html#reading-and-writing-files
Specifically for Windows file systems you will need to do one of the following:
1.) Use forwardslashes vs backslashes. This should work with most OSes.
f = open("c:/somedir/somefile.txt", "r")
2.) Use a raw string.
f = open(r"c:\somedir\somefile.txt", "r")
3.) Escape the backslashes.
f = open("c:\\somedir\\somefile.txt", "r")
If you need to import a module to use in your program from outside your programs directory you can use the below information.
Python looks in the sys.path to see if the module exists there and if so does the import. If the path where you files/modules are located is not in the sys.path, Python will raise an ImportError. You can update the path programmatically by using the sys module.
import sys
dir = "path to mymodule"
if dir not in sys.path:
sys.path.append(dir)
import mymodule
You can check the current sys.path by using:
print(sys.path)
Example:
>>> print(sys.path)
['', '/Library/Frameworks/Python.framework/Versions/3.4/lib/python34.zip', '/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4', '/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/plat-darwin', '/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload', '/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages']
>>> sys.path.append("/Users/ddrummond/pymodules")
>>> print(sys.path)
['', '/Library/Frameworks/Python.framework/Versions/3.4/lib/python34.zip', '/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4', '/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/plat-darwin', '/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload', '/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages', '/Users/ddrummond/pymodules']
>>>
You can see that sys.path now contains '/Users/ddrummond/pymodules'.
I have been trying to get over my fear of Cython (fear because I literally know NOTHING about c, or c++)
I have a function which takes 2 arguments, a set (we'll call it testSet), and a list of sets (we'll call that targetSets). The function then iterates through targetSets, and computes the length of the intersection with testSet, adding that value to a list, which is then returned.
Now, this isn't by itself that slow, but the problem is I need to do simulations of the testSet (and a large number at that, ~ 10,000), and the targetSet is about 10,000 sets long.
So for a small number of simulations to test, the pure python implementation was taking ~50 secs.
I tried making a cython function, and it worked and it's now running at ~16 secs.
If there is anything else that I could do to the cython function that anyone could think of that would be great (python 2.7 btw)
Here is my Cython implementation in overlapFunc.pyx
def computeOverlap(set testSet, list targetSets):
cdef list obsOverlaps = []
cdef int i, N
cdef set overlap
N = len(targetSets)
for i in range(N):
overlap = testSet & targetSets[i]
if len(overlap) <= 1:
obsOverlaps.append(0)
else:
obsOverlaps.append(len(overlap))
return obsOverlaps
and the setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [Extension("overlapFunc",
["overlapFunc.pyx"])]
setup(
name = 'computeOverlap function',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
and some code to build some random sets for testing and to time the function. test.py
import numpy as np
from overlapFunc import computeOverlap
import time
def simRandomSet(n):
for i in range(n):
simSet= set(np.random.randint(low=1, high=100, size=50))
yield simSet
if __name__ == '__main__':
np.random.seed(23032014)
targetSet = [set(np.random.randint(low=1, high=100, size=50)) for i in range(10000)]
simulatedTestSets = simRandomSet(200)
start = time.time()
for i in simulatedTestSets:
obsOverlaps = computeOverlap(i, targetSet)
print time.time()-start
I tried changing the def at the start of the computerOverlap function, as in:
cdef list computeOverlap(set testSet, list targetSets):
but I get the following warning message when I run the setup.py script:
'__pyx_f_11overlapFunc_computeOverlap' defined but not used [-Wunused-function]
and then when I run something that tries to use the function I get an import Error:
from overlapFunc import computeOverlap
ImportError: cannot import name computeOverlap
Thanks in advance for your help,
Cheers,
Davy
In the following line, the extension module name and the filename does not match actual filename.
ext_modules = [Extension("computeOverlapWithGeneList",
["computeOverlapWithGeneList.pyx"])]
Replace it with:
ext_modules = [Extension("overlapFunc",
["overlapFunc.pyx"])]