An object has been serialized by pickle, and it will be used by a model which would be placed at anyplace (under any directory). Since the object is frequently used and kind of a part of the model, I want to have the model contain the pickle file (place the file under a directory of the model) as a variable.
./data/constant.py
object = pickle.load(open('object.pkl'))
./data/object.pkl
./code/model01.py
from ..data import constant
# or
# from __future import absolute_import
# from model.data import constant
object = constant.object
./code/model02.py
from ..data import constant
object = constant.object
The problem is obviously that python will search object.pkl under ./code/(and anywhere I use the function of the model outside of the model) rather than ./data/.
Am I doing it right? Any better solutions? Thanks.
I think this question may be duplicated (this is a very common issue) but I cannot find any related archive here. If so, please help redirect me there.
Doing a little bit of path manipulation should work:
In module constant.py:
import os
path = os.path.dirname(os.path.abspath(__file__))
obj = pickle.load(open(os.path.join(path, 'object.pkl')))
Looks like you want object to be part of the module constant.
One way would be just putting the pickled object in constant.py:
my_object = pickle.loads(pickled_object) # don't use the name `object` it is a built-in
Note the s in loads.
pickled_object needs to be placed inside constant.py before the line shown above. It has to be a byte string.
You can create it either directly from the object:
pickled_object = pickle.dumps(obj)
or take it form the pickled file and past it in.
Example
Pickle your object:
>>> import pickle
>>> obj = [1, 2, 3]
>>> pickle.dumps(obj)
b'\x80\x03]q\x00(K\x01K\x02K\x03e.'
Now, in constant.py:
pickled_object = b'\x80\x03]q\x00(K\x01K\x02K\x03e.'
my_object = pickle.loads(pickled_object)
As a result, my_object is [1, 2, 3] and can be accessed via constant.my_object
Related
I try to process a series of DNG raw picture files and it all works well for the first pass (first fils). When I try to read the second DNG file during the second pass through the for-next loop, I receive the error message 'rawpy._rawpy.RawPy' object has no attribute 'imread' when executing the line "with raw.imread(file) as raw:".
import numpy as np
import rawpy as raw
import pyexiv2
from scipy import stats
for file in list:
metadata = pyexiv2.ImageMetadata(file)
metadata.read()
with raw.imread(file) as raw:
rgb16 = raw.postprocess(gamma=(1,1), no_auto_bright=True, output_bps=16)
avgR=stats.describe(np.ravel(rgb16[:,:,0]))[2]
avgG=stats.describe(np.ravel(rgb16[:,:,1]))[2]
avgB=stats.describe(np.ravel(rgb16[:,:,2]))[2]
print i,file,'T=', metadata['Exif.PentaxDng.Temperature'].raw_value,'C',avgR,avgG,avgB
i+=1
I tried already to close the raw object but from googling I understand that is not necessary when a context manager is used.
Help or suggestions are very welcome.
Thanks in advance.
You're overwriting your alias of the rawpy module (raw) with the image you're reading. That means you'll get an error on the second pass through the loop.
import rawpy as raw # here's the first thing named "raw"
#...
for file in list:
#...
with raw.imread(file) as raw: # here's the second
#...
Pick a different name for one of the variables and your code should work.
I'm basically attempting py2/3 compatibility by trying to add a a fullmatch method to Python2 re compiled patterns.
I've seen https://stackoverflow.com/a/2982/2694385 on how to add an attribute to an existing instance.
I'm creating the method as given in https://stackoverflow.com/a/30212799/2694385.
Example code
import re
import six
regex = re.compile('some pattern') # Better if we keep this same
if six.PY2:
def fullmatch(self_regex, string, flags=0):
return self_regex.match(string, flags=flags)
regex = re.compile(r'(?:' + regex.pattern + r')\Z', flags=regex.flags)
import types
bound = types.MethodType(fullmatch, regex)
# AttributeError here. None of the following three lines work
regex.fullmatch = bound
regex.__setattr__('fullmatch', bound)
setattr(regex, 'fullmatch', bound)
That ain't gonna work - regex objects are created on the C side and they don't represent your regular instances so you cannot modify their signature from within Python. For example, if you try to extend _sre.SRE_Pattern:
import _sre
class CustomPattern(_sre.SRE_Pattern): pass
You'll get an AttributeError complaining that such object doesn't exist in the _sre module. If you try to cheat it with:
import re
tmp_pattern = re.compile("")
sre_pat = type(tmp_pattern)
class CustomPattern(sre_pat): pass
it will give you a TypeError complaining that _sre.SRE_Pattern (which now 'temporarily' exists as it's being created ad-hoc) is not an acceptable base type.
What you can do instead is to create a full wrapper around the re module (or at least add the missing structures to it) and handle the Python version differences on the Python's side, although I think it's just not worth it.
P.S. If you're not using six anywhere else, there no reason for the overhead just to check your Python version - you can use sys.version_info.major < 3 instead.
See nlpia.regexes.Pattern for something similar to what you want -- a Frankenstein mashup of _sre.Pattern with a fullmatch() method. This monkey-patching "inheritance" approach works in Python 2 and 3.
import re
import regex
class Pattern:
""" "Inherits" _sre.SRE_Pattern and adds .fullmatch() method
>>> pattern = Pattern('Aaron[ ]Swartz')
>>> pattern.match('Aaron Swartz')
<_sre.SRE_Match object; span=(0, 12), match='Aaron Swartz'>
>>> pattern.fullmatch('Aaron Swartz!!')
>>> pattern.match('Aaron Swartz!!')
<_sre.SRE_Match object; span=(0, 12), match='Aaron Swartz'>
"""
def __init__(self, pattern):
self._compiled_pattern = re.compile(pattern)
for name in dir(self._compiled_pattern):
if not name in set(('__class__', '__init__', 'fullmatch')):
attr = getattr(self._compiled_pattern, name)
setattr(self, name, attr)
def fullmatch(self, *args, **kwargs):
return regex.fullmatch(self._compiled_pattern.pattern, *args, **kwargs)
I create a tensorflow model which I would like to save to file so that I can predict against it later. In particular, I need to save the:
input_placeholder
(= tf.placeholder(tf.float32, [None, iVariableLen]))
solution_space
(= tf.nn.sigmoid(tf.matmul(input_placeholder, weight_variable) + bias_variable))
session
(= tf.Session())
I've tried using pickle which works on other objects like sklearn binarizers etc, but not on the above, for which I get the error at the bottom.
How I pickle:
import pickle
with open(sModelSavePath, 'w') as fiModel:
pickle.dump(dModel, fiModel)
where dModel is a dictionary that contains all the objects I want to persist, which I use for fitting against.
Any suggestions on how to pickle tensorflow objects?
Error message:
pickle.dump(dModel, fiModel)
...
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle module objects
The way I solved this was by pickleing Sklearn objects like binarizers, and using tensorflow's inbuilt save functions for the actual model:
Saving tensorflow model:
1) Build the model as you usually would
2) Save the session with tf.train.Saver(). For example:
oSaver = tf.train.Saver()
oSess = oSession
oSaver.save(oSess, sModelPath) #filename ends with .ckpt
3) This saves all available variables etc in that session to their variable names.
Loading tensorflow model:
1) The entire flow needs to be re-initialized. In other words, variables, weights, bias, loss function etc need to be declared, and then initialized with tf.initialize_all_variables() being passed into oSession.run()
2) That session now needs to be passed to the loader. I abstracted the flow, so my loader looks like this:
dAlg = tf_training_algorithm() #defines variables etc and initializes session
oSaver = tf.train.Saver()
oSaver.restore(dAlg['oSess'], sModelPath)
return {
'oSess': dAlg['oSess'],
#the other stuff I need from my algorithm, like my solution space etc
}
3) All objects you need for prediction need to be gotten out of your initialisation, which in my case sit in dAlg
PS: Pickle like this:
with open(sSavePathFilename, 'w') as fiModel:
pickle.dump(dModel, fiModel)
with open(sFilename, 'r') as fiModel:
dModel = pickle.load(fiModel)
You should save your project into two separate parts, one is for tensorflow's objects, another is for other objects. I recommend you to use the following tools:
tf.saved_model: the procedures your want to saved and load tensorflow all in it.
dill: a more powerful pickle tool based on pickle, it can help you bypass most errors encountered by pickle
Can't find a direct, head on answer to this. Is there a way to access a tempfile in Django across 2 distinct views? Say I have the following code:
view#1(request):
temp = tempfile.NamedTemporaryFile()
write_book.save(temp_file)
temp_file_name = temp_file.name
print temp_file_name
request.session['output_file_name'] = temp_file_name
request.session.modified = True
return #something or other
view#2(request):
temp_file_name = request.session['output_file_name']
temp_file = open(str(temp_file_name))
#do something with 'temp_file' here
My problem comes in specifically on view#2, the 2nd line "open(temp_file_name)". Django complains this file/pathway doesn't exist, which is consistent of my understanding of the tempfile module (that the file is 'hidden' and only available to Django).
Is there a way for me to access this file? In case it matters, I ONLY need to read from it (technically serve it for download).
I'd think of this as how to access a NamedTemporaryFile across different requests, rather than different views. Looking at this documentation on NamedTemporaryFile, it says that the file can be opened across the same process, but not necessarily across multiple processes. Perhaps your other view is being called in a different Django process.
My suggestion would be to abandon the use of NamedTemporaryFile and instead just write it as a permanent file, then delete the file in the other view.
Thanks seddonym for attempting to answer. My partner clarified this for me...seddonym is correct for the Django version of NamedTemporaryFile. By calling the python version (sorry, don't have enough cred to post hyperlinks. Stupid rule) you CAN access across requests.
The trick is setting the delete=False parameter, and closing the file before 'returning' at the end of the request. Then, in the subsequent request, just open(file_name). Psuedo code below:
>>> import tempfile
>>> file = tempfile.NamedTemporaryFile(delete=False)
>>> file.name
'c:\\users\\(blah)\(blah)\(blah)\\temp\\tmp9drcz9'
>>> file.close()
>>> file
<closed file '<fdopen>', mode 'w+b' at 0x00EF5390>
>>> f = open(file.name)
>>> f
<open file 'c:\users\ymalik\appdata\local\temp\tmp9drcz9', mode 'r' at 0x0278C128>
This is, of course, done in the console, but it works in django as well.
I have a list including 4000 elements in python which each of its elements is an object of following class with several values.
class Point:
def __init__(self):
self.coords = []
self.IP=[]
self.BW=20
self.status='M'
def __repr__(self):
return str(self.coords)
I do not know how to save this list for future uses.
I have tried to save it by open a file and write() function, but this is not what I want.
I want to save it and import it in next program, like what we do in MATLAB that we can save a variable and import it in future
pickle is a good choice:
import pickle
with open("output.bin", "wb") as output:
pickle.dump(yourList, output)
and symmetric:
import pickle
with open("output.bin", "rb") as data:
yourList = pickle.load(data)
It is a good choice because it is included with the standard library, it can serialize almost any Python object without effort and has a good implementation, although the output is not human readable. Please note that you should use pickle only for your personal scripts, since it will happily load anything it receives, including malicious code: I would not recommend it for production or released projects.
This might be an option:
f = open('foo', 'wb')
np.save(f, my_list)
for loading then use
data = np.load(open('foo'))
if 'b' is not present in 'wb' then the program gives an error:
TypeError: write() argument must be str, not bytes
"b" for binary makes the difference.
Since you say Matlab, numpy should be an option.
f = open('foo', 'w')
np.save(f, my_list)
# later
data = np.load(open('foo'))
Of course, it'll return an array, not a list, but you can coerce it if you really want an array...