I am getting this output in the interactive mode.
class test:
def p(self):
print 'PP'
>>> f=open('E:\Python\Roy Progs\Test','w')
>>> t=test()
>>> import pickle
>>> pickle.dump(t,f)
>>> f.close()
>>> f=open('E:\Python\Roy Progs\Test','r')
>>> pickle.load(f).p()
PP
>>> f.close()
>>>
=============================== RESTART: Shell ===============================
>>> f=open('E:\Python\Roy Progs\Test','r')
>>> import pickle
>>> pickle.load(f).p()
Traceback (most recent call last):
File "<pyshell#14>", line 1, in <module>
pickle.load(f).p()
File "E:\Python\lib\pickle.py", line 1384, in load
return Unpickler(file).load()
File "E:\Python\lib\pickle.py", line 864, in load
dispatch[key](self)
File "E:\Python\lib\pickle.py", line 1075, in load_inst
klass = self.find_class(module, name)
File "E:\Python\lib\pickle.py", line 1132, in find_class
klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'test'
From the output I realize that the definition of the class (whose object is stored in the file) must be in there in the RAM at the time of retrieving data and using it. I however do not understand why this must be the case, by storing objects in the file am I not storing the class definition also?
The pickle module stores classes by named reference. If you change the name or location of the class pickle will raise an error.
A quick illustration of that can be seen in the interactive:
>>> class test:
x = 5
>>> from pickle import dumps
>>> dumps(test)
'c__main__\ntest\np0\n.' # pickle is storing a reference to 'test'
To successfully call load pickle must be able to find the previously defined class (which is destroyed when you call restart in idle)
Related
I have a date of birth (dob) DateField in my Django Model (Author). I tried to annotate age parameter. I searched for many possible ways to do it and each procedure generated some kind of error.
Here I tried in python console first to make sure that the expression would be a valid one:
>>> from datetime import datetime
>>> (datetime.now() - datetime(2000,1,1)).days #output: 7506
First Try:
>>> from django.db.models import F
>>> authors = Author.objects.annotate(age = (datetime.now()-F('dob')).days) #no-error here
>>> print(authors) # Exception thrown here
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/puru/Documents/Python/django/prac1/django-master/lib/python3.7/site-packages/Django-3.2-py3.7.egg/django/db/models/query.py", line 324, in __getitem__
qs._fetch_all()
File "/home/puru/Documents/Python/django/prac1/django-master/lib/python3.7/site-packages/Django-3.2-py3.7.egg/django/db/models/query.py", line 1305, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/home/puru/Documents/Python/django/prac1/django-master/lib/python3.7/site-packages/Django-3.2-py3.7.egg/django/db/models/query.py", line 70, in __iter__
for row in compiler.results_iter(results):
File "/home/puru/Documents/Python/django/prac1/django-master/lib/python3.7/site-packages/Django-3.2-py3.7.egg/django/db/models/sql/compiler.py", line 1100, in apply_converters
value = converter(value, expression, connection)
File "/home/puru/Documents/Python/django/prac1/django-master/lib/python3.7/site-packages/Django-3.2-py3.7.egg/django/db/backends/sqlite3/operations.py", line 291, in convert_datefield_value
value = parse_date(value)
File "/home/puru/Documents/Python/django/prac1/django-master/lib/python3.7/site-packages/Django-3.2-py3.7.egg/django/utils/dateparse.py", line 75, in parse_date
match = date_re.match(value)
TypeError: expected string or bytes-like object
Second time I used ExpressionWrapper to define that output type will be DateTimeField
>>> from django.db.models import DateTimeField, ExpressionWrapper
>>> authors = Author.objects.annotate(age = ExpressionWrapper(timezone.now() - F('dob'), output_field=DateTimeField()).days)
AttributeError: 'ExpressionWrapper' object has no attribute 'days'
I have also tried the RawSQL. I am using Sqlite database here. so date('now') would provide me current date.
>>> from django.db.models.expressions import RawSQL
>>> authors = Author.objects.annotate(age=RawSQL("date('now')-dob"))
Traceback (most recent call last):
File "<console>", line 1, in <module>
TypeError: __init__() missing 1 required positional argument: 'params'
So is there any way, I could solve the issue?
You were almost there, the issue was the output_field value. It should be DurationField instead of DateTimeField
age_expr = ExpressionWrapper(timezone.now() - F('dob'), output_field=DateTimeField())
queryset = Author.objects.annotate(age=age_expr)
NOTE: You can't use .days along with ExpressionWrapper since the annotate operation performs in the DB level.
To calculate the age, you may need to use the .total_seconds() method
print(queryset[0].age.total_seconds() / 60 / 60 / 24 / 365.25)
Empty string or list params worked for the RawSQL command.
authors = Author.objects.annotate(age=RawSQL("date('now')-dob", params=""))
I'm working on a project with several packages and sub-packages and got stuck with a import problem:
Traceback (most recent call last):
File "***/src/dev/dev_salle.py", line 5, in <module>
from src.envir.jeu import Salle
File "***/src/envir/jeu/__init__.py", line 1, in <module>
from camera import Camera
File "***/src/envir/jeu/camera.py", line 5, in <module>
class Camera(object):
TypeError: Error when calling the metaclass bases
'function' object is not iterable
From what I understand, it have to do with metaclasses of Camera and other classes derivating from object conflicting because of nested import. Some other things I noticed:
The error stays if I instantiate Camera from camera.py as main
if __name__ == "__main__":
print Camera()
The error disappears when I delete the object inheritance, however I want to keep it to use __slots__.
The error is still there if I create a metaclass MetaCamera(type) for Camera in camera.py
The error does not occur if I call
class Blabla(object):pass
if __name__ == "__main__":
print Blabla()
in ***/src/, ***/src/envir/ or ***/src/envir/jeu/
I also import other classes from another subpackage, Pti(object) and Porte(Pti). Pti import is just fine but Porte raise this error:
Traceback (most recent call last):
File "***/src/dev/dev_salle.py", line 5, in <module>
from src.envir.jeu import Salle
File "***/src/envir/jeu/__init__.py", line 2, in <module>
from salle import Salle
File "***/src/envir/jeu/salle.py", line 10, in <module>
from src.envir.elements.pti import Pti, Porte, Monnaie
File "***/src/envir/elements/pti.py", line 14, in <module>
class Porte(Pti):
TypeError: Error when calling the metaclass bases
'function' object is not iterable
A recap of the structure because it's a bit messy:
src
|--dev
| '--dev_salle.py
'--envir
|--jeu
| |--camera.py
| '--salle.py
'--elements
'--pti.py
I'm used to neither metaclasses nor nested imports so I'm a bit at a lost there...
I finally solve my problem!
And that have nothing to do with nested imports --'
In the two classes whose import raise the error, I defined __slots__ with:
def __slots__(self): return ("bla", "bli", "blu")
while the correct definition is:
__slots__ = ("bla", "bli", "blu")
Since the metaclass use __slots__ to initiate the class dictionnary, it raise a metaclass error.
I am new to Python and as my first project I am attempting to convert a Python2 script to Python3.
The script is failing when it attempts to serialize a class using pickle.
It seems as though it is failing as I am trying to save a class which uses the Cmd CLI.
This code works using Python2.
Can anyone tell me what is wrong with the script and how I fix it?
import sys
import cmd
try:
import pickle as pickle
except:
import pickle
import os.path
def main():
app = Labyrinth()
turnfile = "turn0.lwot"
app.Save(turnfile)
class CLI(cmd.Cmd):
def __init__(self):
cmd.Cmd.__init__(self)
class Labyrinth(cmd.Cmd):
def __init__(self):
cmd.Cmd.__init__(self)
def Save(self, fname):
with open(fname, 'wb') as f:
pickle.dump(self,f, 2)
f.close()
print ("Save Successful!")
sys.exit()
if __name__ == '__main__':
main()
Not all objects are picklable. In particular, file objects are problematic because you can't generally restore their state later. cmd.Cmd holds stdin and stdout file objects and that should make them unpicklable. I was quite surprised that it worked in python 2, but it didn't really... Even though the stdin and stdout pickled, the unpickled object you get back later doesn't work, as in this example:
>>> import sys
>>> import pickle
>>> sys.stdout.write('foo\n')
foo
>>> serialized = pickle.dumps(sys.stdout, 2)
>>> stdout = pickle.loads(serialized)
>>> stdout.write('bar\n')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: I/O operation on closed file
>>>
So, even though this bit of code didn't fail, the object shouldn't be usable later. You can add a few special methods to an object that let you fix objects so they can be serialized. Here, I've stripped the bad attributes on save and added them back on restore. Now you can pickle, unpickle and it actually works when you are done.
import sys
import cmd
try:
import cPickle as pickle
except:
import pickle
import os.path
def main():
app = Labyrinth()
turnfile = "turn0.lwot"
app.Save(turnfile)
class CLI(cmd.Cmd):
def __init__(self):
cmd.Cmd.__init__(self)
class Labyrinth(cmd.Cmd):
def __init__(self):
cmd.Cmd.__init__(self)
def Save(self, fname):
with open(fname, 'wb') as f:
pickle.dump(self,f, pickle.HIGHEST_PROTOCOL)
f.close()
print ("Save Successful!")
sys.exit()
def __getstate__(self):
# stdin/out are unpicklable. We'll get new ones on load
return tuple(((k,v) for k,v in self.__dict__.items()
if k not in ('stdin', 'stdout')))
def __setstate__(self, state):
self.__dict__.update(state)
self.stdin = sys.stdin
self.stdout = sys.stdout
if __name__ == '__main__':
main()
Playing with the protocol doesn't help. The full error message (which you should have included) is:
1027:~/mypy$ python3 stack41334887.py
Traceback (most recent call last):
File "stack41334887.py", line 33, in <module>
main()
File "stack41334887.py", line 14, in main
app.Save(turnfile)
File "stack41334887.py", line 27, in Save
pickle.dump(self,f, 3, fix_imports=True)
TypeError: cannot serialize '_io.TextIOWrapper' object
Python3 made some major changes in the io system. This TextIOWrapper is, I think new to Py3.
https://docs.python.org/3.1/library/io.html#io.TextIOWrapper
Can I use multiprocessing.Pool in a method of a class? also had problems serializing a TextIOWrapper.
=========
So inspireed by #tdelaney, I checked the stdin for my PY3 session:
In [1212]: sys.stdin
Out[1212]: <_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>
So that's the thing that can't be serialized.
Trying to serialize numpy array with Pyro4 is returning the following type error
TypeError: don't know how to serialize class <type 'numpy.ndarray'>. Give it vars() or an appropriate __getstate__
The code is a the following
import numpy as np
import Pyro4
# set pickle serializer
Pyro4.config.SERIALIZERS_ACCEPTED = set(['pickle','json', 'marshal', 'serpent'])
#Pyro4.expose
class test(object):
def get_array(self):
return np.random.random((10,10))
def main():
# create a Pyro daemon
daemon = Pyro4.Daemon()
# register test instance
uri = daemon.register(test())
# print uri to connect to it in another console
print uri
# start the event loop of the server to wait for calls
daemon.requestLoop()
if __name__=="__main__":
main()
now open another console and try to call test instance doing the following
import Pyro4
Pyro4.config.SERIALIZERS_ACCEPTED = set(['pickle','json', 'marshal', 'serpent'])
# connect to URI which is printed above
# must be something like this 'PYRO:obj_c261949088104b839878255b98a9da90#localhost:57495'
p = Pyro4.Proxy(URI)
p.get_array()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/Pyro4/core.py", line 171, in __call__
return self.__send(self.__name, args, kwargs)
File "/usr/local/lib/python2.7/site-packages/Pyro4/core.py", line 438, in _pyroInvoke
raise data
TypeError: don't know how to serialize class <type 'numpy.ndarray'>. Give it vars() or an appropriate __getstate
This is mentioned in the manual including what you can do to solve it: http://pythonhosted.org/Pyro4/tipstricks.html#pyro-and-numpy
In your code above, you didn't tell your client code to use pickle. You should use another config item for that (SERIALIZER). What you have there is meant for Pyro deamons instead.
I have a manager in my django project and am implementing a filter as below.
I start the django shell and get this error:
>>> from django.http import HttpRequest
>>> r=HttpRequest()
>>> r.session=()
>>> from movierating.models import *
>>> RatingModel.statManager.RatingTimeLine(r)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/data/dashapp/movierating/managers.py", line 27, in RatingTimeLine
(data,display,err)=self.getdatacache(request)
File "/data/dashapp/movierating/managers.py", line 11, in getdatacache
filterDict=request.session.get('RatingFilter',{})
AttributeError: 'tuple' object has no attribute 'get'
>>> from django.http import HttpRequest
>>> r=HttpRequest()
>>> r.session()
Traceback (most recent call last):
File "<console>", line 1, in <module>
AttributeError: 'HttpRequest' object has no attribute 'session'r code here
The file manager.py looks like this:
import json,datetime
from django.db import models
from django.db.models import *
from pandas import *
from urllib import urlencode
import hashlib
from django.core.cache import cache
from web.extras import *
class RatingManager(models.Manager):
def getdatacache(self,request):
filterDict=request.session.get('RatingFilter',{})
(fapply,display,err)=normalizeFilter(self.model,filterDict)
cache_id=urlencode(fapply.items())
cache_id=hashlib.md5('RatingFilter'+cache_id).hexdigest()
data=None
if data==None:
res=self.model.objects.order_by('date').filter(**fapply).values('date').annotate(
RatingCounts = Count('rating'),
RatingSum = Sum('rating'),
)
data=DataFrame(list(res))
data['AverageRating']=data['RatingSum']/data['RatingCounts']
cache.set(cache_id,data)
return (data,display,err)
def RatingTimeLine(self,request):
jsondata={}
jsondata['chartconfig']={}
jsondata['chartconfig']['title']="Average Movie Rating per Day"
jsondata['chartconfig']['names']=['AverageRating']
(data,display,err)=self.getdatacache(request)
jsondata['chartconfig']['errors']="<br/>".join(err)
jsondata['chartconfig']['subtitle']="<br/>".join(display)
jsondata['series']=data[['data','AverageRating']].values.tolist()
data=json.dumps(jsondata,cls = SeriesEncoder)
return {'data':data}
I have this model in my models.py:
class RatingModel(models.Model):
movie=models.ForeignKey(MovieModel)
user=models.ForeignKey(userModel)
rating=models.IntegerField()
date=models.DateField()
objects=models.Manager() #default manager
statManager=RatingManager() #new manager class
# FilterMapping={
#'movie':'movie__name', #django relational
# }
def __unicode__(self):
return "id: %d rating for movie is %s" %(self.id,self.movie.name) #relationships
class Meta:
app_label = 'movierating'
db_table = 'rating'
What could be the possible error in this line?
filterDict=request.session.get('RatingFilter',{})
In your third line you say r.session=() - that is, you're assigning an empty tuple to r.session. Your code later tries to call get() on this tuple, which isn't a supported operation.
I'm not sure what you're trying to do here. request.session is normally filled in by the session middleware. If you're trying to test your code programmatically, you might want to look into the Django testing framework, which supports sessions.