I've installed the Tensorflow package and complied it from source for IBM s390x architecture. The image recognition classify_image.py sample as described in the tutorial throws an error as shown below:
Run command:
python ./classify_image.py --model_dir=/data/shared/myprojects/tensorflow/models/models-master/tutorials/image/imagenet --image_file=/data/shared/myprojects/keras/images/claude_profile.jpg
Error message:
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 659, in _call_cpp_shape_fn_impl
raise ValueError(err.message)
ValueError: Cannot reshape a tensor with 1041082757314414592 elements to shape [16777216,524288] (8796093022208 elements) for 'pool_3/_reshape' (op: 'Reshape') with input shapes: [1,22546423,22546423,2048], [2] and with input tensors computed as partial shapes: input[1] = [16777216,524288].
Version:
python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.VERSION
'1.3.1'
>>>
A possible error cause is an incompatibility of endianness as the trained model likely is stored in a little endian notation while the CPU works in a big endian mode. Is there an easy way to configure a byte swapping that changes the endianness of the input data? Other Tensorflow samples, without image processing routines execute OK.
1041082757314414592 sounds more like an overflow / underflow than endianness issue. If you don't try to load an example but try to run one from scratch do you also see issues?
This seems to be happening because the inception model being a pre-trained model on a Little Endian machine, will have issues while loading on Big Endian(s390x). Also any graph (eg. classify_image_graph_def.pb) will store values like size in one format, which gives unexpected results when read in the other.
As far as I know there is no tool available yet to convert any saved model to be compatible on big endian.
So for now, on big endian, we need to train our nodes from scratch.
Related
I'm working on the crash dump of a standard C++ program.
I'm having a problem with following piece of code:
_variant_t* pvarTemp;
pvarTemp = <static_cast><_variant_t*> (pointer_array->GetAt(i));
if (!pvarTemp) continue;
pvarTemp->Clear();
As I presume something's going wrong in memory, I decided to have a look at the memory window (menu "Debug", "Windows", "Memory", "Memory 1"), where I entered the memory address of pvarTemp, but I only saw some question marks.
The crash dump itself is just a small crashdump, hereby PYKD's targetSystem().desc result (in Windgb):
0:049> !py
Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> print targetSystem().desc
User mini dump: C:\Temp_Folder\CRASH.DMP
My question is: what's the reason of the question marks?
The reason for having the question marks also is the reason for the application to crash.
As I'm working with a small crashdump, the memory addresses, mentioned in the dump, might not be trustworthy, so the question marks are normal behaviour.
Which one of the two is correct?
You're doing postmortem debugging. Generally, crash dumps don't capture all the memory in use by the process. If you try to inspect memory that wasn't included in the dump, you'll see question marks.
If you're doing live debugging, question marks mean the address isn't mapped into the process space.
I have a function built that reads in hex-ascii encoded data, I built that in Python 2.7. I am changing my code over to run on 3.x and hit an unforeseen issue. The function worked flawlessly under 2.7. Here is what I have:
# works with 2.7
data = open('hexascii_file.dat', 'rU').read()
When I run that under 3.x I get a UnicodeError:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x85 in position 500594: invalid start byte
I thought the default codec under Python 2.7 was ascii, so I tried the following under 3.x:
data = open('hexascii_file.dat', 'rU', encoding='ascii')
This did not work (same error as above, but specifying 'ascii' instead of 'utf-8'. However, when I use the latin-1 codec all works well.
data = open('hexascii_file.dat', 'rU', encoding='latin-1')
I guess I am looking for a quick sanity check here to ensure I have made the proper change to the script. Does this change make sense?
Two uServices are communicating via a message queue (RabbitMQ). The data is encoded using message pack.
I have the following scenarios:
python3 -> python3: working fine
python2 -> python3: encoding issues
Encoding is done with:
umsgpack.packb(data)
Decoding with:
umsgpack.unpackb(body)
When doing encoding and decoding in python3 I get:
data={'sender': 'producer-big-red-tiger', 'json': '{"msg": "hi"}', 'servicename': 'echo', 'command': 'run'}
When doing encoding in python2 and decoding on python3 I get:
data={b'command': b'run', b'json': b'{"msg": ""}', b'servicename': b'echo', b'sender': b'bla-blah'}
Why is the data non "completely" decoded? What should I do on the sender / receiver to achieve compatibility between python2 and python3?
Look at the "Notes" section of the README from msgpack-python;
msgpack can distinguish string and binary type for now. But it is not like Python 2. Python 2 added unicode string. But msgpack renamed raw to str and added bin type. It is because keep compatibility with data created by old libs. raw was used for text more than binary.
Currently, while msgpack-python supports new bin type, default setting doesn't use it and decodes raw as bytes instead of unicode (str in Python 3).
You can change this by using use_bin_type=True option in Packer and encoding="utf-8" option in Unpacker.
>>> import msgpack
>>> packed = msgpack.packb([b'spam', u'egg'], use_bin_type=True)
>>> msgpack.unpackb(packed, encoding='utf-8')
['spam', u'egg']
I have a Python module that is operating as a server for a wireless handheld computer. Every time the handheld sends a message to the server, the module determines what kind of message it is, and then assembles an appropriate response. Because the responses are often state-dependent, I am using global variables where needed to retain/share information between the individual functions that handle each type of message.
The problem I'm having is when the application is closed (for whatever reason), the global variable values are (of course) lost, so on re-launching the application it's out of synch with the handheld. I need a reliable way to store those values for recovery.
The direction I've gone so far (but have not gotten it to work yet) is to write the variable names and their values to a CSV file on the disk, every time they're updated -- and then (when the app is launched), look for that file and use it to assign the variables to their previous states. I have no trouble writing the file or reading it, but for some reason the values just aren't getting assigned.
I can post the code for comments/help, but before that I wanted to find out whether I'm just going an entirely wrong direction in the first place. Is there a better (or at least preferable) way to save and recover these values?
thanks,
JDM
====
Following up. It may be a touch klunky, but here's what I have and it's working. The only globals I care about are the ones that start with "CUR_". I had to use tempDict1 because the interpreter doesn't seem to like iterating directly over globals().
import pickle
CUR_GLO1 = 'valglo1'
CUR_GLO2 = 'valglo2'
CUR_GLO3 = 'valglo3'
def saveGlobs():
tempDict1 = globals().copy()
tempDict2 = {}
for key in tempDict1:
if (key[:4]=='CUR_'):tempDict2[key] = tempDict1[key]
pickle.dump(tempDict2,open('tempDict.p','wb'))
def retrieveGlobs():
tempDict = pickle.load(open('tempDict.p','rb'))
globals().update(tempDict)
writing it up as an answer..
What I think you want to do is a form of application checkpointing.
You can use the Pickle module for conveniently saving and loading Python variables. Here is a simple example of how to use it. This discussion on Stackoverflow and this note seem to agree, although part of me thinks that there must be a better way.
Incidentally, you don't need to put everything into a dictionary. As long as you dump and load variables in the right order, and make sure that you don't change that, insert data in the middle etc, you can just dump and load several variables. Using a dictionary like you did does remove the ordering dependency though.
% python
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> foo=123
>>> bar="hello"
>>> d={'abc': 123, 'def': 456}
>>> f=open('p.pickle', 'wb')
>>> pickle.dump(foo, f)
>>> pickle.dump(bar, f)
>>> pickle.dump(d, f)
>>> ^D
% python
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> f=open('p.pickle','rb')
>>> foo=pickle.load(f)
>>> foo
123
>>> bar=pickle.load(f)
>>> bar
'hello'
>>> d=pickle.load(f)
>>> d
{'abc': 123, 'def': 456}
>>>
Does anyone know of some CRC test vectors for CRC16-CCITT?
I do not have a CRC implementation I can trust and either need to test someone's implementation or my own. (For CRC32, I use the PNG code as the gold standard, as it's a reputable reference implementation.)
(this site's CRC calculator looks useful but I need to verify correctness somehow)
UPDATE: The above CRC calculator looks useful but it takes only ascii, no way to enter hex. --- it's very awkward to enter hex input, though. (ASCII 12 in hex can be entered as %31%32, so you can't just copy+paste a long string of hexadecimal bytes; also the % character doesn't seem to have an escape)
I have verified this online calculator, which takes hex inputs, against the Boost test vectors for CRC16, CRC16-CCITT, and CRC32.
Boost has a nice CRC implementation you can test against. As far as I know it's possible to configure it for CRC16.
http://www.boost.org/doc/libs/1_41_0/libs/crc/index.html
There seems to be an example of how to set it up to simulate CCITT on this page: http://www.boost.org/doc/libs/1_41_0/libs/crc/crc.html
Python's binascii package has had CRC-16 for a while.
Use binascii.crc_hqx(val, 0xFFFF) - the previous example are...
$ python3
Python 3.7.3 (default, Dec 20 2019, 18:57:59)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii
>>> tv = binascii.a2b_hex("12345670")
>>> hex(binascii.crc_hqx(tv, 0xFFFF))
'0xb1e4'
>>> tv = "123456789".encode("ascii")
>>> hex(binascii.crc_hqx(tv, 0xFFFF))
'0x29b1'
Here are two test vectors for CCITT-16 CRC (whose polynomial is X16 + X12 + X5 + 1 (0x1021 in big-endian hex representation); initial CRC value is 0xFFFF. XOR value out is zero.) are:
0x12345670 = 0xB1E4
0x5A261977 = 0x1AAD
I've found this one:
http://introcs.cs.princeton.edu/java/51data/CRC16CCITT.java.html
"123456789".getBytes("ASCII"); -> 0x29b1