Is tf.Variable a tensor or not? - unit-testing

I've read some answers on this question here and here, however I'm still a bit puzzled by tf.Variable being and/or not being a tf.Tensor.
The linked answers deal with a mutability of tf.Variable and mentioning that tf.Variables maintains their states (when instantiated with default parameter trainable=True).
What makes me still a bit confused is a test case I came across when writing simple unit tests using tf.test.TestCase
Consider the following code snippet. We have a simple class called Foo which has only one property, a tf.Variable initialized to w:
import tensorflow as tf
import numpy as np
class Foo:
def __init__(self, w):
self.w = tf.Variable(w)
Now, let's say you want to test that the instance of Foo has w initialized with tensor of the same dimension as passed in via w. The simplest test case could be written as follows:
import tensorflow as tf
import numpy as np
from foo import Foo
class TestFoo(tf.test.TestCase):
def test_init(self):
w = np.random.rand(3,2)
foo = Foo(w)
init = tf.global_variables_initializer()
with self.test_session() as sess:
sess.run(init)
self.assertShapeEqual(w, foo.w)
if __name__ == '__main__':
tf.test.main()
Now when you run the test you'll get the following error:
======================================================================
ERROR: test_init (__main__.TestFoo)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_foo.py", line 12, in test_init
self.assertShapeEqual(w, foo.w)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/test_util.py", line 1100, in assertShapeEqual
raise TypeError("tf_tensor must be a Tensor")
TypeError: tf_tensor must be a Tensor
----------------------------------------------------------------------
Ran 2 tests in 0.027s
FAILED (errors=1)
You can "get around" this unit test error by doing something like this (i.e. note assertShapeEqual was replaced with assertEqual):
self.assertEqual(list(w.shape), foo.w.get_shape().as_list())
What I'm interested in, though, is the tf.Variable vs tf.Tensor relationship.
What the test error seems to be suggesting is that foo.w is NOT a tf.Tensor, meaning you probably can't use tf.Tensor API on it. Consider, however, the following interactive python session:
$ python3
Python 3.6.3 (default, Oct 4 2017, 06:09:15)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> import numpy as np
>>> w = np.random.rand(3,2)
>>> var = tf.Variable(w)
>>> var.get_shape().as_list()
[3, 2]
>>> list(w.shape)
[3, 2]
>>>
In the session above, we create a variable and run the get_shape() method on it to retrieve its shape dimensions. Now, get_shape() method is a tf.Tensor API method as you can see here.
So to get back to my question, what parts of tf.Tensor API does tf.Variable implement. If the answer is ALL of them, why does the above test case fail?
self.assertShapeEqual(w, foo.w)
with
raise TypeError("tf_tensor must be a Tensor")
I'm pretty sure I'm missing something fundamental here or maybe it's a bug in assertShapeEqual ? I would appreciate if someone could shed some light on this.
I'm using following version of tensorflow on macOS with python3:
tensorflow (1.4.1)

That testing utility function is checking whether a variable implements tf.Tensor
>>> import tensorflow as tf
>>> v = tf.Variable('v')
>>> v
<tf.Variable 'Variable:0' shape=() dtype=string_ref>
>>> isinstance(v, tf.Tensor)
False
The answer appears to be 'no'.
Update:
According to the documentation that is correct:
https://www.tensorflow.org/programmers_guide/variables
Unlike tf.Tensor objects, a tf.Variable exists outside the context of
a single session.run call.
Although:
A tf.Variable represents a tensor whose value can be changed by
running ops on it.
(Not quite sure what 'represents a tensor' means - sounds like a design 'feature')

Related

Why is pytesseract causing AttributeError: 'NoneType' object has no attribute 'bands'?

I am trying to get started using pytesseract but as you can see below I am having problems.
I have found people getting what seems to be the same error and they say that it is a bug in PIL 1.1.7. Others say the problem is caused by PIL being lazy and one needs to force PIL to load the image with im.load() after opening it, but that didn't seem to help. Any suggestions gratefully received.
K:\Glamdring\Projects\Images\OCR>python
Python 2.7.8 (default, Jun 30 2014, 16:03:49) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from PIL import Image
>>> import pytesseract
>>> pytesseract.image_to_string(Image.open('foo.png'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build\bdist.win32\egg\pytesseract\pytesseract.py", line 143, in image_to_string
File "c:\Python27_32\lib\site-packages\PIL\Image.py", line 1497, in split
if self.im.bands == 1:
AttributeError: 'NoneType' object has no attribute 'bands'
Try to use objects from Image and pytesseract module separately.
It solved my problem:
try:
import Image
except ImportError:
from PIL import Image
import pytesseract
img = Image.open('myImage.jpg')
img.load()
i = pytesseract.image_to_string(img)
print i
I have no prior experience with PIL but I was bored so I tried to look into it and, from what I can tell, it is probably a bug.
This isn't a fault of pytesseract if we look at the execution steps.
Initially your Image.open('foo.png') works perfectly fine with no errors relating to your stack-trace.
pytesseract.image_to_string(img) comes in afterwards and does the following:
# Omitting the rest of the method.
# calls method split() of your image object.
if len(image.split()) == 4:
This is the first statement acting on image so we know we have to look back into PIL to find the root of the problem.
Your stacktrace has the specific message AttributeError: 'NoneType' object has no attribute 'bands' with regard to the if self.im.bands statement. This means that im is the object = None.
Lets look into the image.split() method:
"""
Split this image into individual bands. This method returns a
tuple of individual image bands from an image. For example,
splitting an "RGB" image creates three new images each
containing a copy of one of the original bands (red, green,
blue).
:returns: A tuple containing bands.
"""
self.load() # This is the culprit since..
if self.im.bands == 1: # .. here the im attribute of the image = None
ims = [self.copy()]
# Omitting the rest ---
Obviously self.load() sets, among others, the im value. I verified this with a test Image and it seemed to work with no issues [I suggest you try the same with your image]:
In [7]: print img.im
None
In [8]: img.load()
Out[8]: <PixelAccess at 0x7fe03ab6a210>
In [9]: print img.im
<ImagingCore object at 0x7fe03ab6a1d0>
Let's now take a look in load():
I don't generally have the knowledge to know the internals here but I did observe something iffy: many FIXME comments before the assignment of im, specifically:
# -- Omitting rest --
# FIXME: on Unix, use PROT_READ etc
self.map = mmap.mmap(file.fileno(), size)
self.im = Image.core.map_buffer(
self.map, self.size, d, e, o, a
)
# -- Omitting rest --
if hasattr(self, "tile_post_rotate"):
# FIXME: This is a hack to handle rotated PCD's
self.im = self.im.rotate(self.tile_post_rotate)
self.size = self.im.size
This might be an indication that there might be some issues needing attention here. I can't be 100% certain though.
Of course, this might be caused by your image for some reason. The load() method worked fine with an image I supplied (and pytesseract just gave me a different error :P). You're better off probably creating a new issue for this. If any PIL experts happen to see this, enlighten us if you can.
im.load() worked for me on running program in administrator mode and also add this line if you don't have tesseract executable in your PATH
pytesseract.pytesseract.tesseract_cmd = 'C:/Program Files (x86)/Tesseract-OCR/tesseract.exe'
If you have already read an image(not using im.load() but with imread()) or frame from video and did some image processing stuff(may be not) on that variable(image) then you need to give the following command
pytesseract.image_to_string(Image.fromarray(image))
As #J_Mascis said,using objects worked here too-
import pytesseract
from PIL import Image
img = Image.open('im.jpg')
img.load()
print(pytesseract.image_to_string(img, lang='eng'))#'eng' for english

Discovering keys using h5py in python3

In python2.7, I can analyze an hdf5 files keys use
$ python
>>> import h5py
>>> f = h5py.File('example.h5', 'r')
>>> f.keys()
[u'some_key']
However, in python3.4, I get something different:
$ python3 -q
>>> import h5py
>>> f = h5py.File('example.h5', 'r')
>>> f.keys()
KeysViewWithLock(<HDF5 file "example.h5" (mode r)>)
What is KeysViewWithLock, and how can I examine my HDF5 keys in Python3?
From h5py's website (http://docs.h5py.org/en/latest/high/group.html#dict-interface-and-links):
When using h5py from Python 3, the keys(), values() and items()
methods will return view-like objects instead of lists. These objects
support containership testing and iteration, but can’t be sliced like
lists.
This explains why we can't view them. The simplest answer is to convert them to a list:
>>> list(for.keys())
Unfortunately, I run things in iPython, and it uses the command 'l'. That means that approach won't work.
In order to actually view them, we need to take advantage of containership testing and iteration. Containership testing means we'd have to already know the keys, so that's out. Fortunately, it's simple to use iteration:
>>> [key for key in f.keys()]
['mins', 'rects_x', 'rects_y']
I've created a simple function that does this automatically:
def keys(f):
return [key for key in f.keys()]
Then you get:
>>> keys(f)
['mins', 'rects_x', 'rects_y']

how to call python functions defined in another .py file without interactive mode

I wrote two scripts: modbus_master.py and modbus_helpers.py.
modbus_helpers.py is just a bunch of raw functions I defined that I'm trying to call from modbus_master.py.
When I try to execute 'modbus_master.py' from the windows CLI this happens...
C:\Python27\modbus_simulator>modbus_master.py
Traceback (most recent call last):
File "C:\Python27\modbus_master.py", line 3, in <module>
import modbus_helpers
ImportError: No module named modbus_helpers
However,
If I go to python interactive mode and do this...
C:\Python27\modbus_simulator>python
Python 2.7.5 (default, May 15 2013, 22:43:36) MSC v.1500 32 bit (Intel) on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import modbus_master
The code in modbus_master.py that calls modbus_helpers.py works just fine. So how do I bridge this gap here so that I can just do this and run the script without error?
C:\Python27\modbus_simulator>modbus_master.py
Code in modbus_master.py:
import sys
import json
import modbus_helpers
import os
def printRegsets():
print 'these register sets were piped in...\r\n'
regsetIndex = 0
for regset in registersetsList:
print str(regsetIndex) , ':', regset['Name']
regsetIndex = regsetIndex + 1
path = os.path.normpath('C:\Python27\modbus_simulator\export2.txt')
registersetsList = modbus_helpers.getRegisterSetFromACMExportFile(path)
printRegsets()
Found the solution to the problem. There was nothing wrong with the code. I missed the obvious...modbus_master.py must be in the same folder/directory as modbus_helpers.py for the 'import' statement to work.

Confused about __import__ in Python

I trying to import module by __import__ like this:
>>> mod = __import__('x.y.z')
But I only got x:
>>> print mod
>>> <module 'x' from '...'>
How should I do to import z ? I tried like this, it works but i don't know why.
>>> mod = __import__('x.y.z', {}, {}, [''])
>>> print mod
>>> <module 'x.y.z' from '...'>
I'm really confused about this, and I also have no idea with the globals and locals parameters.
Thx a lot!
Relevant notes from the docs (__import__):
When the name variable is of the form package.module, normally, the top-level package (the name up till the first dot) is returned, not the module named by name. However, when a non-empty fromlist argument is given, the module named by name is returned.
Hence, it's similar to writing import x.y.z which also makes x available (as well as x.y and x.y.z).
Use the importlib module instead. Of which the bare bones are made available in 2.7.
import importlib
z = importlib.import_module("z", "x.y")
# equivalent to
from x.y import z

Simple matplotlib Annotating example not working in Python 2.7

Code
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure()
ax = fig.add_subplot(111)
t = np.arange(0.0, 5.0, 0.01)
s = np.cos(2*np.pi*t)
line, = ax.plot(t, s, lw=2)
ax.annotate('local max', xy=(2, 1), xytext=(3, 1.5),
arrowprops=dict(facecolor='black', shrink=0.05),
)
ax.set_ylim(-2,2)
plt.show()
from http://matplotlib.org/1.2.0/users/annotations_intro.html
return
TypeError: 'dict' object is not callable
I manged to fixed it with
xxx={'facecolor':'black', 'shrink':0.05}
ax.annotate('local max', xy=(2, 1), xytext=(3, 1.5),
arrowprops=xxx,
)
Is this the best way ?
Also what caused this problem ? ( I know that this started with Python 2.7)
So if somebody know more, please share.
Since the code looks fine and runs ok on my machine, it seems that you may have a variable named "dict" (see this answer for reference). A couple of ideas on how to check:
use Pylint.
if you suspect one specific builtin, try checking it's type (type(dict)), or look at the properties/functions it has (dir(dict)).
open a fresh notebook and try again, if you only observe the problem in interactive session.
try alternate syntax to initialise the dictionary
ax.annotate('local max', xy=(2, 1), xytext=(3, 1.5),
arrowprops={'facecolor':'black', 'shrink':0.05})
try explicitly instancing a variable of this type, using the alternate syntax (as you did already).