Python csv writer "AttributeError: __exit__" issue - python-2.7

I have three variables I want to write in a tab delimited .csv, appending values each time the script iterates over a key value from the dictionary.
Currently the script calls a command, regex the stdout as out then assigns the three defined regex groups to individual variables for writing to .csv labeled first second and third. I get a __exit_ error when I run the below script.
/note I've read up on csv.writer and I'm still confused as to whether I can actually write multiple variables to a row.
Thanks for any help you can provide.
import csv, re, subprocess
for k in myDict:
run_command = "".join(["./aCommand", " -r data -p ", str(k)])
process = subprocess.Popen(run_command,
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out, err = process.communicate()
errcode = process.returncode
pattern = re.compile('lastwrite|(\d{2}:\d{2}:\d{2})|alert|trust|Value')
grouping = re.compile('(?P<first>.+?)(\n)(?P<second>.+?)([\n]{2})(?P<rest>.+[\n])',
re.MULTILINE | re.DOTALL)
if pattern.findall(out):
match = re.search(grouping, out)
first = match.group('first')
second = match.group('second')
rest = match.group('rest')
with csv.writer(open(FILE, 'a')) as f:
writer = csv.writer(f, delimiter='\t')
writer.writerow(first, second, rest)
Edit: Requested in the comments to post entire traceback, note the line listed in traceback will not match the above code as this is not the entire script.
Traceback (most recent call last):
File "/mydir/pyrr.py", line 60, in <module>
run_rip()
File "/mydir/pyrr.py", line 55, in run_rip
with csv.writer(open('/mydir/ntuser.csv', 'a')) as f:
AttributeError: __exit__
Answer: Using the below comment I was able to write it as follows.
f = csv.writer(open('/mydir/ntuser.csv', 'a'),
dialect=csv.excel,
delimiter='\t')
f.writerow((first, second, rest))

The error is pretty clear. The with statement takes a context manager, i.e., an object with an __enter__ and an __exit__ method, such as the object returned by open. csv.writer does not provide such an object. You are also attempting to create the writer twice:
with open(FILE, 'a') as f:
writer = csv.writer(f, delimiter='\t')
writer.writerow(first, second, rest)
The with ... f: is like a try...except...finally that guarantees that f is closed no matter what happens, except you don't have to type it out. open(...) returns a context manager whose __exit__ method is called in that finally block you don't have to type. That is what your exception was complaining about. open returns an object that has __exit__ properly defined and can therefore handle normal exit and exceptions in the with block. csv.writer does not have such a method, so you can't use it in the with statement itself. You have to do it in the with block following the statement, as I've shown you.

Related

Getting TypeError trying to open() a file in write mode with Python

I have a Python script that in my mind should:
Open a file
Save its content in a variable
For each line in the variable:
Edit it with a regular expression
Append it to another variable
Write the second variable to the original file
Here is a MWE version of the script:
# [omitting some setup]
with open(setFile, 'r') as setFile:
olddata = setFile.readlines()
newdata = ''
for line in olddata:
newdata += re.sub(regex, newset, line)
with open(setFile, 'w') as setFile:
setFile.write(newdata)
When I run the script I get the following error:
Traceback (most recent call last):
File C:\myFolder\myScript.py, line 11, in <module>
with open(setFile, 'w') as setFile:
TypeError: expected str, bytes or os.PathLike object, not _io.TextIOWrapper
As far as I can understand, Python is complaining about receiving the setFile variable as an argument of open() because it isn't of the expected type, but then why did it accept it before (when I only read the file)?
I suppose that my mistake is quite evident but, as I am a neophyte in Python, I can't find out where it is. Could anyone give me a help?
just curious why you are using the same variable name for your file and then as your filehandler and then again in your next with function.
_io.TextIOWrapper is the object from your previous open, which has been asssigned to the setFile variable.
try:
with open(setFile, 'r') as readFile:
olddata = readFile.readlines()
newdata = ''
for line in olddata:
newdata += re.sub(regex, newset, line)
with open(setFile, 'w') as writeFile:
writeFile.write(newdata)

Saving files in Python using "with" method

I am wanting to create a file and save it to json format. Every example I find specifies the 'open' method. I am using Python 2.7 on Windows. Please help me understand why the 'open' is necessary for a file I am saving for the first time.
I have read every tutorial I could find and researched this issue but with no luck still. I do not want to create the file outside of my program and then have my program overwrite it.
Here is my code:
def savefile():
filename = filedialog.asksaveasfilename(initialdir =
"./Documents/WorkingDirectory/",title = "Save file",filetypes = (("JSON
files","*.json"), ("All files", "*.")))
with open(filename, 'r+') as currentfile:
data = currentfile.read()
print (data)
Here is this error I get:
Exception in Tkinter callback Traceback (most recent call last):
File "C:\Python27\lib\lib-tk\Tkinter.py", line 1542, in call
return self.func(*args) File "C:\Users\CurrentUser\Desktop\newproject.py", line 174, in savefile
with open(filename, 'r+') as currentfile: IOError: [Errno 2] No such file or directory:
u'C:/Users/CurrentUser/Documents/WorkingDirectory/test.json'
Ok, I figured it out! The problem was the mode "r+". Since I am creating the file, there is no need for read and write, just write. So I changed the mode to 'w' and that fixed it. I also added the '.json' so it would be automatically added after the filename.
def savefile():
filename = filedialog.asksaveasfilename(initialdir =
"./Documents/WorkingDirectory/",title = "Save file",filetypes = (("JSON
files","*.json"), ("All files", "*.")))
with open(filename + ".json", 'w') as currentfile:
line1 = currentfile.write(stringone)
line2 = currentfile.write(stringtwo)
print (line1,line2)

Python Celery group() - TypeError: [...] argument after ** must be a mapping, not long

I'm trying to run a celery (3.1.17) task that executes further tasks in a group but I always run into errors. This is how I set up the code:
from celery import task, group
#task
def daily_emails():
[...]
all_tasks = []
for chunk in range(0, users.count(), 1000):
some_users = users[chunk:chunk+1000]
all_tasks.append(write_email_bunch.subtask(some_users, execnum))
job = group(all_tasks)
# result = job.apply_async()
# job.get()
result = job.delay()
print result
results = result.join()
print results
print "done writing email tasks"
count = sum(results)
print count
#task
def write_email_bunch(some_users, execnum):
[...]
return len(some_users) - skipped_email_count
And this is the output:
<GroupResult: 3d766c85-21af-4ed0-90cb-a1ca2d281db1 [69527252-8468-4358-9328-144f727f372b, 6d03d86e-1b69-4f43-832e-bd27c4dfc092, 1d868d1b-b502-4672-9895-430089e9532e]>
Traceback (most recent call last):
File "send_daily_emails.py", line 8, in <module>
daily_emails()
File "/var/www/virtualenvs/nt_dev/local/lib/python2.7/site-packages/celery/app/task.py", line 420, in __call__
return self.run(*args, **kwargs)
File "/var/www/nt_dev/nt/apps/emails/tasks.py", line 124, in daily_emails
results = result.join()
File "/var/www/virtualenvs/nt_dev/local/lib/python2.7/site-packages/celery/result.py", line 642, in join
interval=interval, no_ack=no_ack,
File "/var/www/virtualenvs/nt_dev/local/lib/python2.7/site-packages/celery/result.py", line 870, in get
raise self.result
TypeError: write_email_bunch() argument after ** must be a mapping, not long
So I get a GroupResult but somehow Im unable to join it or further process it.
And when I use write_email_bunch.s(some_users, execnum) I get this exception:
File "/var/www/virtualenvs/nt_dev/local/lib/python2.7/site-packages/celery/result.py", line 870, in get
raise self.result
TypeError: 'tuple' object is not callable
How would I wait for all the Group Tasks to be completed to continue afterwards?
job.get() gives me this exception:
TypeError: get expected at least 1 arguments, got 0
subtask takes a tuple of args, a dict of kwargs and task options so it should be called like this:
all_tasks.append(write_email_bunch.subtask((some_users, execnum)))
note that we are passing it a tuple containing the args
Also you shouldn't wait on a task inside a task - this can cause deadlocks. In this case I reckon daily_emails does not need to be a celery task - it can be a regular function that creates a canvas object and runs apply async.
def daily_emails():
all_tasks = []
for chunk in range(0, users.count(), 1000):
some_users = users[chunk:chunk+1000]
all_tasks.append(write_email_bunch.subtask(some_users, execnum))
job = group(all_tasks)
result = job.apply_async()
return result.id
In addition to the other answer you could be using chunks here:
http://docs.celeryproject.org/en/latest/userguide/canvas.html#chunks
#app.task
def daily_emails():
return write_email.chunks(users, 1000).delay()
#task
def write_email(user):
[...]
It may be beneficial to do it manually if getting several objects at once
from the db is important. You should also consider that the model objects will be serialized here, to avoid that you can send the pk only and refetch the model in the task, or send the fields that you care about (like email address or whatever is required to send that email to the user).

How to store the triples in 4store

File "<console>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/django_gstudio-0.3.dev-py2.7.egg/gstudio/testing1.py", line 129, in rdf_description
store.add(self,(subject, predicate, object),context)
File "/usr/local/lib/python2.7/dist-packages/rdflib-3.2.0-py2.7.egg/rdflib/plugins/memory.py", line 298, in add
Store.add(self, triple, context, quoted)
File "/usr/local/lib/python2.7/dist-packages/rdflib-3.2.0-py2.7.egg/rdflib/store.py", line 177, in add
def add(self, (subject, predicate, object), context, quoted=False):
in
store.add(self, (subject, predicate, object), context, quoted=False)
AFAIK - rdflib does not support 4store. But you can easily assert the triples using curl and python and the 4store SPARQL Server. Here there is an example:
import subprocess
command = ["curl","-s",
"-T","/some/file/with/triples",
"-H","Content-Type: application/x-turtle",
"http://localhost:port/data/http://graph.to/save/triples"]
p = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
output, err = p.communicate()
ret = p.poll()
if ret <> 0:
raise Exception, "Error asserting triples"
In this example the content type is turtle but you can use any of the other RDF serializations (ntriples, rdfxml).
If you do not want to deal with subprocesses you can also translate this call into a urllib/urllib2 function.
There are more examples in the 4store SparqlServer documentation. And, optionally, you can use any of the Python 4store client libraries.

django cnotes not working

I have just installed django-cnotes
But it wont work.
It just throws up this error
Traceback (most recent call last):
File "/Library/Python/2.5/site-packages/django/core/servers/basehttp.py", line 279, in run
self.result = application(self.environ, self.start_response)
File "/Library/Python/2.5/site-packages/django/core/servers/basehttp.py", line 651, in __call__
return self.application(environ, start_response)
File "/Library/Python/2.5/site-packages/django/core/handlers/wsgi.py", line 245, in __call__
response = middleware_method(request, response)
File "/Library/Python/2.5/site-packages/django_cnote-0.3.4-py2.5.egg/cnotes/middleware.py", line 47, in process_response
signed_data = self.sign('cnotes', base64.urlsafe_b64encode(Pickle.dumps(cnotes.cnotes)))
PicklingError: Can't pickle <class 'django.utils.functional.__proxy__'>: attribute lookup django.utils.functional.__proxy__ failed
And it is not even in the normal django error debug page. What you see above is all there is on the screen.
And I have just used it as described on github, I just dont get it. Any one have an idea for what is causing this?
UPDATE:
Okay, so I have found something, I think.
message = _("You have successfully altered ")
message += edituser.username
cnotes.add(message)
message2 = _("You may now close ")
cnotes.add(message2)
This will cause the error. So I thought "Okay, I can only call it once per view" That would have been stupid and it was indeed not the cause.
The following code will produce no error
message = _("You have successfully altered ")
message += edituser.username
cnotes.add(message)
message2 = '_("You may now close ")'
cnotes.add(message2)
But is not because of the translation it uses that fine just 2 lines above, but it has to be something with doing another translation or something. Im lost.
It appears as though pickle is receiving an object of type django.utils.functional.__proxy__. This means either your input is weird, or there is a bug in cnotes.
If there is something wrong with your input to cnotes, you should see it if you take a look at the types of your messages (I used the manage.py shell):
>>> message = _("You have successfully altered ")
>>> message += "Bob Knoblick"
>>> type(message)
<type 'unicode'>
>>> message2 = _("You may now close ")
>>> type(message2)
<type 'unicode'>
>>>
If your types come back as anything other than unicode or str, I'd dig into your code and figure out where that other type is coming from, or ensure that it can be pickled.
If there is something wrong within cnotes, you should get the same error doing this:
cnotes.add(u'Foo')
cnotes.add(u'Bar')
cnotes.add(u'Baz')
Per the original author:
The translated string, _("You may now close ") was not ending up as a unicode string. One can use this to force unicode before sending to cnotes:
message2 = unicode(_("You may now close "))