I have a list including 4000 elements in python which each of its elements is an object of following class with several values.
class Point:
def __init__(self):
self.coords = []
self.IP=[]
self.BW=20
self.status='M'
def __repr__(self):
return str(self.coords)
I do not know how to save this list for future uses.
I have tried to save it by open a file and write() function, but this is not what I want.
I want to save it and import it in next program, like what we do in MATLAB that we can save a variable and import it in future
pickle is a good choice:
import pickle
with open("output.bin", "wb") as output:
pickle.dump(yourList, output)
and symmetric:
import pickle
with open("output.bin", "rb") as data:
yourList = pickle.load(data)
It is a good choice because it is included with the standard library, it can serialize almost any Python object without effort and has a good implementation, although the output is not human readable. Please note that you should use pickle only for your personal scripts, since it will happily load anything it receives, including malicious code: I would not recommend it for production or released projects.
This might be an option:
f = open('foo', 'wb')
np.save(f, my_list)
for loading then use
data = np.load(open('foo'))
if 'b' is not present in 'wb' then the program gives an error:
TypeError: write() argument must be str, not bytes
"b" for binary makes the difference.
Since you say Matlab, numpy should be an option.
f = open('foo', 'w')
np.save(f, my_list)
# later
data = np.load(open('foo'))
Of course, it'll return an array, not a list, but you can coerce it if you really want an array...
Related
I try to process a series of DNG raw picture files and it all works well for the first pass (first fils). When I try to read the second DNG file during the second pass through the for-next loop, I receive the error message 'rawpy._rawpy.RawPy' object has no attribute 'imread' when executing the line "with raw.imread(file) as raw:".
import numpy as np
import rawpy as raw
import pyexiv2
from scipy import stats
for file in list:
metadata = pyexiv2.ImageMetadata(file)
metadata.read()
with raw.imread(file) as raw:
rgb16 = raw.postprocess(gamma=(1,1), no_auto_bright=True, output_bps=16)
avgR=stats.describe(np.ravel(rgb16[:,:,0]))[2]
avgG=stats.describe(np.ravel(rgb16[:,:,1]))[2]
avgB=stats.describe(np.ravel(rgb16[:,:,2]))[2]
print i,file,'T=', metadata['Exif.PentaxDng.Temperature'].raw_value,'C',avgR,avgG,avgB
i+=1
I tried already to close the raw object but from googling I understand that is not necessary when a context manager is used.
Help or suggestions are very welcome.
Thanks in advance.
You're overwriting your alias of the rawpy module (raw) with the image you're reading. That means you'll get an error on the second pass through the loop.
import rawpy as raw # here's the first thing named "raw"
#...
for file in list:
#...
with raw.imread(file) as raw: # here's the second
#...
Pick a different name for one of the variables and your code should work.
I create a tensorflow model which I would like to save to file so that I can predict against it later. In particular, I need to save the:
input_placeholder
(= tf.placeholder(tf.float32, [None, iVariableLen]))
solution_space
(= tf.nn.sigmoid(tf.matmul(input_placeholder, weight_variable) + bias_variable))
session
(= tf.Session())
I've tried using pickle which works on other objects like sklearn binarizers etc, but not on the above, for which I get the error at the bottom.
How I pickle:
import pickle
with open(sModelSavePath, 'w') as fiModel:
pickle.dump(dModel, fiModel)
where dModel is a dictionary that contains all the objects I want to persist, which I use for fitting against.
Any suggestions on how to pickle tensorflow objects?
Error message:
pickle.dump(dModel, fiModel)
...
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle module objects
The way I solved this was by pickleing Sklearn objects like binarizers, and using tensorflow's inbuilt save functions for the actual model:
Saving tensorflow model:
1) Build the model as you usually would
2) Save the session with tf.train.Saver(). For example:
oSaver = tf.train.Saver()
oSess = oSession
oSaver.save(oSess, sModelPath) #filename ends with .ckpt
3) This saves all available variables etc in that session to their variable names.
Loading tensorflow model:
1) The entire flow needs to be re-initialized. In other words, variables, weights, bias, loss function etc need to be declared, and then initialized with tf.initialize_all_variables() being passed into oSession.run()
2) That session now needs to be passed to the loader. I abstracted the flow, so my loader looks like this:
dAlg = tf_training_algorithm() #defines variables etc and initializes session
oSaver = tf.train.Saver()
oSaver.restore(dAlg['oSess'], sModelPath)
return {
'oSess': dAlg['oSess'],
#the other stuff I need from my algorithm, like my solution space etc
}
3) All objects you need for prediction need to be gotten out of your initialisation, which in my case sit in dAlg
PS: Pickle like this:
with open(sSavePathFilename, 'w') as fiModel:
pickle.dump(dModel, fiModel)
with open(sFilename, 'r') as fiModel:
dModel = pickle.load(fiModel)
You should save your project into two separate parts, one is for tensorflow's objects, another is for other objects. I recommend you to use the following tools:
tf.saved_model: the procedures your want to saved and load tensorflow all in it.
dill: a more powerful pickle tool based on pickle, it can help you bypass most errors encountered by pickle
I am trying to gather information on files being opened and closed so I can keep track of which processes is doing what. I work with users that will not systematically use custom I/O functions and methods so I would like to change the behavior of builtins.
I figured out a way to change the open function. It might not be optimal, but it works.
import logging
import os
import sys
import io
import logging
import __builtin__
def custom_open(file_name, access_mode, buffering = 0):
logging.debug('Opening file ' + file_name)
return __builtin__.open(file_name, access_mode, buffering)
logging.getLogger('').setLevel(logging.DEBUG)
open = custom_open
f = open('workfile', 'w')
f.write('This is a test\n')
f.close()
What I would now like to do is change the behavior of the file close method. I tried a few things but nothing works.
To elaborate a little on my comment, since nobody else seems to be submitting an answer:
# Create a custom file class
class custom_file(file):
def close(self):
# do logging
super(file, self).close()
# create a custom file opener
def custom_open(*args, **kwargs):
# do logging
return custom_file(*args, **kwargs)
# Set local `open` to point to your custom_open fn
open = custom_open
Implicit here (and I didn't do research so it could be wrong) is that open('bla') is just calling out to file.__init__('bla'), so be careful there.
edit: You might also want to make sure you're overriding other methods on file, like flush or anything else that would cause Python to touch your disk.
Can't find a direct, head on answer to this. Is there a way to access a tempfile in Django across 2 distinct views? Say I have the following code:
view#1(request):
temp = tempfile.NamedTemporaryFile()
write_book.save(temp_file)
temp_file_name = temp_file.name
print temp_file_name
request.session['output_file_name'] = temp_file_name
request.session.modified = True
return #something or other
view#2(request):
temp_file_name = request.session['output_file_name']
temp_file = open(str(temp_file_name))
#do something with 'temp_file' here
My problem comes in specifically on view#2, the 2nd line "open(temp_file_name)". Django complains this file/pathway doesn't exist, which is consistent of my understanding of the tempfile module (that the file is 'hidden' and only available to Django).
Is there a way for me to access this file? In case it matters, I ONLY need to read from it (technically serve it for download).
I'd think of this as how to access a NamedTemporaryFile across different requests, rather than different views. Looking at this documentation on NamedTemporaryFile, it says that the file can be opened across the same process, but not necessarily across multiple processes. Perhaps your other view is being called in a different Django process.
My suggestion would be to abandon the use of NamedTemporaryFile and instead just write it as a permanent file, then delete the file in the other view.
Thanks seddonym for attempting to answer. My partner clarified this for me...seddonym is correct for the Django version of NamedTemporaryFile. By calling the python version (sorry, don't have enough cred to post hyperlinks. Stupid rule) you CAN access across requests.
The trick is setting the delete=False parameter, and closing the file before 'returning' at the end of the request. Then, in the subsequent request, just open(file_name). Psuedo code below:
>>> import tempfile
>>> file = tempfile.NamedTemporaryFile(delete=False)
>>> file.name
'c:\\users\\(blah)\(blah)\(blah)\\temp\\tmp9drcz9'
>>> file.close()
>>> file
<closed file '<fdopen>', mode 'w+b' at 0x00EF5390>
>>> f = open(file.name)
>>> f
<open file 'c:\users\ymalik\appdata\local\temp\tmp9drcz9', mode 'r' at 0x0278C128>
This is, of course, done in the console, but it works in django as well.
I didn't find quite what I was looking for.
I want to obtain the output (stdout) from a python function in real time.
The actual problem is that I want to plot a graph (with cplot from sympy) with a progress bar in my UI. The argument verbose makes cplot output the progress to stdout.
sympy.mpmath.cplot(lambda z: z, real, imag, verbose=True)
The output would be something like:
0 of 71
1 of 71
2 of 71
...
And so on.
I want to capture line by line so I can make a progress bar. (I realize this might not be possible without implementing multithreading). I'm using python2.7 (mainly because I need libraries that aren't in python3)
So, ¿How do I achieve that?
You can capture stdout by monkeypatching sys.stdout. A good way to do it is using a context manager, so that it gets put back when you are done (even if the code raises an exception). If you don't use a context manager, be sure to put the original sys.stdout back using a finally block.
You'll need an object that is file-like, that takes the input and does what you want with it. Subclassing StringIO is a good start. Here's an example of a context manager that captures stdout and stderr and puts them in the result of the bound variable.
class CapturedText(object):
pass
#contextmanager
def captured(disallow_stderr=True):
"""
Context manager to capture the printed output of the code in the with block
Bind the context manager to a variable using `as` and the result will be
in the stdout property.
>>> from tests.helpers import capture
>>> with captured() as c:
... print('hello world!')
...
>>> c.stdout
'hello world!\n'
"""
import sys
stdout = sys.stdout
stderr = sys.stderr
sys.stdout = outfile = StringIO()
sys.stderr = errfile = StringIO()
c = CapturedText()
yield c
c.stdout = outfile.getvalue()
c.stderr = errfile.getvalue()
sys.stdout = stdout
sys.stderr = stderr
if disallow_stderr and c.stderr:
raise Exception("Got stderr output: %s" % c.stderr)
(source)
It works as shown in the docstring. You can replace StringIO() with your own class that writes the progress bar.
Another possibility would be to monkeypatch sympy.mpmath.visualization.print, since cplot uses print to print the output, and it uses from __future__ import print_function.
First, make sure you are using from __future__ import print_function if you aren't using Python 3, as this will otherwise be a SyntaxError.
Then something like
def progressbar_print(*args, **kwargs):
# Take *args and convert it to a progress output
progress(*args)
# If you want to still print the output, do it here
print(*args, **kwargs)
sympy.mpmath.visualization.print = progressbar_print
You might want to monkeypatch it in a custom function that puts it back, as other functions in that module might use print as well. Again, remember to do this using either a context manager or a finally block so that it gets put back even if an exception is raised.
Monkeypatching sys.stdout is definitely the more standard way of doing this, but I like this solution in that it shows that having print as a function can actually be useful.