Python: Mock Opening File, return Actual File - python-2.7

I need to test a call to gzip.open, but I need it to supply it with an actual test file with test data in it to read. I've seen several very similar questions, but none of them work as expected.
This is the code I'm testing:
with gzip.open(local_file_path,'r') as content:
for line in content:
try:
if line.startswith('#'):
continue
line_data = line.split('\t')
request_key = line_data[LINE_FORMAT['date']]
request_key += '-' + line_data[LINE_FORMAT['time']][:-3]
request_key += '-' + line_data[LINE_FORMAT['source_ip']]
if request_key in result.keys():
result[request_key] += 1
else:
result[request_key] = 1
num_requests += 1
except Exception, e:
print ("[get_outstanding_requesters] \t\tError to process line: %s"%line)
I think the problem is related to the issues discussed here because the code treats the file as an iterator, but none of the workarounds discussed have worked for me.
I've tried variations on this:
test_data = open('test_access_log').read()
m = mock.mock_open(read_data=test_data)
m.return_value.__iter__ = lambda self:self
m.return_value.__next__ = lambda self: self.readline()
with mock.patch('gzip.open', m):
with gzip.open('asdf') as f:
for i in f:
print i
Which results in:
TypeError: iter() returned non-iterator of type 'MagicMock'
I'm using Python 2.7. I'm tearing my hair out over this one. Is my only solution to forget trying to use an iterator (actual files can be very large, which is why I'm trying to avoid doing so?)

This is working:
import unittest
import mock
test_data = open('test_access_log').read()
m = mock.mock_open(read_data=test_data)
m.return_value.__iter__.return_value = test_data.splitlines()
with mock.patch('gzip.open', m):
with gzip.open('test_access_log') as f:
for i in f:
print i
Thanks to bash-shell.net

Related

Official ZeroOut gradient example error: AttributeError: 'list' object has no attribute 'eval'

I followed the official tutorial of the tensorflow website: https://www.tensorflow.org/extend/adding_an_op
There is also described how to call the gradient of the example ZeroOut in the tutorial that I want to try in this short code snippet underneath.
I have found the code here: https://github.com/MatteoRagni/tf.ZeroOut.gpu
import numpy as np
import tensorflow as tf
from tensorflow.python.framework import ops
from tensorflow.python.ops import array_ops
from tensorflow.python.ops import sparse_ops
zero_out_module = tf.load_op_library('./libzeroout.so')
#ops.RegisterGradient("ZeroOut")
def _zero_out_grad(op, grad):
to_zero = op.inputs[0]
shape = array_ops.shape(to_zero)
index = array_ops.zeros_like(shape)
first_grad = array_ops.reshape(grad, [-1])[0]
to_zero_grad = sparse_ops.sparse_to_dense([index], shape, first_grad, 0)
return [to_zero_grad] # List of one Tensor, since we have one input
t_in = tf.placeholder(tf.int32, [None,None])
ret = zero_out_module.zero_out(t_in)
grad = tf.gradients(ys=tf.reduce_sum(ret), xs=t_in)
with tf.Session(''):
feed_dict = {t_in: [[1, 2], [3, 4]]}
print "ret val: ", ret.eval(feed_dict=feed_dict)
print "grad: ", grad
print "grad: ", grad.eval(feed_dict=feed_dict)
I got this error ...
AttributeError: 'list' object has no attribute 'eval'
... but I can do ret.eval().
Why I cant call grad.eval()? I want to see these values inside the grad tensor. How to debug gradient?
Answer to old question
The implementation
def _zero_out_grad(op, *grads):
topdiff = grads[0]
bottom = op.inputs[0]
shape = array_ops.shape(bottom)
index = array_ops.zeros_like(shape)
first_grad = array_ops.reshape(topdiff, [-1])[0]
to_zero_grad = sparse_ops.sparse_to_dense([index], shape, first_grad, 0)
return to_zero_grad
works quite nicely here. Are you sure "#ops.RegisterGradient("ZeroOut")" is executed before the tf.Session()?
Usually the
zero_out_module = tf.load_op_library('./libzeroout.so')
#ops.RegisterGradient("ZeroOut")
def _zero_out_grad(op, grad):
# ...
is placed in a different file and just imported. A full working example even with the recent TensorFlow version is here.
Answer to completely changed question
Your gradient function returns a list and a Python list has no 'eval()'. Try either:
grad = tf.gradients(ys=tf.reduce_sum(ret), xs=t_in)[0]
Or follow best practice and use
grad = tf.gradients(ys=tf.reduce_sum(ret), xs=t_in)
with tf.Session() as sess:
sess.run(grad, feed_dict=feed_dict)
Please do not change your entire question

Creating a Dictionary from a while loop (Python)

How can I create a Dictionary from my while loop below? infile reads from a file called input, which has the following content:
min:1,23,62,256
max:24,672,5,23
sum:22,14,2,3,89
P90:23,30,45.23
P70:12,23,24,57,32
infile = open("input.txt", "r")
answers = open("output.txt", "w")
while True:
line = infile.readline()
if not line: break
opType = line[0:3]
numList = (line[4:len(line)])
numList = numList.split(',')
What I'm trying to do is basically 2 lists, one that has the operation name (opType) and the other that has the numbers. From there I want to create a dictionary that looks like this
myDictionary = {
'min': 1,23,62,256,
'max': 24,672,5,23,
'avg': 22,14,2,3,89,
'P90': 23,30,45.23,
'P70': 12,23,24,57,32,
}
The reason for this is that I need to call the operation type to a self-made function, which will then carry out the operation. I'll figure this part out. I currently just need help making the dictionary from the while loop.
I'm using python 2.7
Try the following code.
I believe, you would need the 'sum' also in the dictionary. If not, just add a condition to remove it.
myDictionary = {}
with open('input.txt','r') as f:
for line in f:
x = line.split(':')[1].rstrip().split(',')
for i in xrange(len(x)):
try:
x[i] = int(x[i])
except ValueError:
x[i] = float(x[i])
myDictionary[line.split(':')[0]] = x
print myDictionary

replacing specific lines in a text file using python

First of all I am pretty new at python, so bear with me. I am attempting to read from one file, retrieve specific values and overwrite old values in another file with a similar format. The format is 'text value=xxx' in both files. I have the first half of the program working, I can extract the values I want and have placed them into a dict named 'params{}'. The part I haven't figured out is how to just write the specific value into the target file without it showing up at the end of the file or just writing garbage or only half of the file. Here is my source code so far:
import os, os.path, re, fileinput, sys
#set the path to the resource files
#res_files_path = r'C:\Users\n518013\Documents\203-104 WA My MRT Files\CIA Data\pelzer_settings'
tst_res_files_path = r'C:\resource'
# Set path to target files.
#tar_files_path = r'C:\Users\n518013\Documents\203-104 WA My MRT Files\CIA Data\CIA3 Settings-G4'
tst_tar_files_path = r'C:\target'
#test dir.
test_files_path = r'C:\Users\n518013\Documents\MRT Equipment - BY 740-104 WA\CIA - AS\Setting Files\305_70R_22_5 setting files\CIA 1 Standard'
# function1 to find word index and point to value
def f_index(lst, item):
ind = lst.index(item)
val = lst[ind + 3]
print val
return val
# function 2 for values only 1 away from search term
def f_index_1(lst, item):
ind = lst.index(item)
val = lst[ind + 1]
return val
# Create file list.
file_lst = os.listdir(tst_res_files_path)
# Traverse the file list and read in dim settings files.
# Set up dict.
params = {}
#print params
for fname in file_lst:
file_loc = os.path.join(tst_res_files_path, fname)
with open(file_loc, 'r') as f:
if re.search('ms\.', fname):
print fname
break
line = f.read()
word = re.split('\W+', line)
print word
for w in word:
if w == 'Number':
print w
params['sectors'] = f_index(word, w)
elif w == 'Lid':
params['lid_dly'] = f_index(word, w)
elif w == 'Head':
params['rotation_dly'] = f_index(word, w)
elif w == 'Horizontal':
tmp = f_index_1(word, w)
param = int(tmp) + 72
params['horizontal'] = str(param)
elif w == 'Vertical':
tmp = f_index_1(word, w)
param = int(tmp) - 65
params['vertical'] = str(param)
elif w == 'Tilt':
params['tilt'] = f_index_1(word, w)
else:
print 'next...'
print params #this is just for debugging
file_tar = os.path.join(tst_tar_files_path, fname)
for lines in fileinput.input(file_tar, inplace=True):
print lines.rstrip()
if lines.startswith('Number'):
if lines[-2:-1] != params['sectors']:
repl = params['sectors']
lines = lines.replace(lines[-2:-1], repl)
sys.stdout.write(lines)
else:
continue
Sample text files:
[ADMINISTRATIVE SETTINGS]
SettingsType=SingleScan
DimensionCode=
Operator=
Description=rev.1 4sept03
TireDimClass=Crown
TireWidth=400mm
[TEST PARAMETERS]
Number Of Sectors=9
Vacuum=50
[DELAY SETTINGS]
Lid Close Delay=3
Head Rotation Delay=3
[HEAD POSITION]
Horizontal=140
Vertical=460
Tilt=0
[CALIBRATION]
UseConvFactors=0
LengthUnit=0
ConvMMX=1
ConvPixelX=1
CorrFactorX=1
ConvMMY=1
ConvPixelY=1
CorrFactorY=1
end sample txt.
The code I have only writes about half of the file back, and I don't understand why? I am trying to replace the line 'Number of Sectors=9' with 'Number of Sectors=8' if I could get this to work, the rest of the replacements can be done using if statements.
Please help! I've spent hours on google looking for answers and info and everything I find gets me close but no cigar!
Thank you all in advance!
your file has the '.ini' format. python supports reading and writing those with the ConfigParser module. you could do this:
# py3: from pathlib import Path
import os.path
import configparser
# py3: IN_PATH = Path(__file__).parent / '../data/sample.ini'
# py3: OUT_PATH = Path(__file__).parent / '../data/sample_out.ini'
HERE = os.path.dirname(__file__)
IN_PATH = os.path.join(HERE, '../data/sample.ini')
OUT_PATH = os.path.join(HERE, '../data/sample_out.ini')
config = configparser.ConfigParser()
# py3: config.read(str(IN_PATH))
config.read(IN_PATH)
print(config['CALIBRATION']['LengthUnit'])
config['CALIBRATION']['LengthUnit'] = '27'
# py3: with OUT_PATH.open('w') as fle:
with open(OUT_PATH, 'w') as fle:
config.write(fle)

'list' object has no attribute 'items'

working on ubuntu 12.04, python 2.7
I have the error 'list' object has no attribute 'items' when I call the print_to_screen function.
could someone explain me please how to manage the dictionary which is created?
Thanks
class median_uniq:
def init(self):
self.median_number_list = []
def print_to_screen(self, words_dict, is_reverse = False):
words = words_dict.items()
words.sort(key = lambda(a,b):(a,b), reverse = is_reverse)
print("[Words tweeted: %d]" % len(words)).center(60,"=")
print("%-25s | %25s" % ("Words", "count"))
print BANNER
for w, c in words:
print("%-25s | %25d" % (w, c))
def Median_number(self, file_name):
file_object = open(file_name, "r")
number_word_list = []
for line in file_object:
unique_words_per_tweet = sorted(set(line.rstrip().split(" ")))
number_word_list.append(len(Counter((unique_words_per_tweet))))
self.median_number_list.append(numpy.median(numpy.array(number_word_list)))
print self.median_number_list
return self.median_number_list
to call function, I do this:
med = median_uniq() med_list = med.Median_number(input_file)
med.print_to_screen(med_list, is_reverse = False)
on command line knowing that my input_file is a .txt file. thanks
I have seen similar post, but don't well understand. An explanation with simply words would be really helpful.
Thanks
"words_dict" in your code is a list, but your attempt to use a python dictionary method (items())
I would need a working example to be able to propose a customized solution.

IndexError, but more likely I/O error

Unsure of why I am getting this error. I'm reading from a file called columns_unsorted.txt, then trying to write to columns_unsorted.txt. There error is on fan_on = string_j[1], saying list index out of range. Here's my code:
#!/usr/bin/python
import fileinput
import collections
# open document to record results into
j = open('./columns_unsorted.txt', 'r')
# note this is a file of rows of space-delimited date in the format <1384055277275353 0 0 0 1 0 0 0 0 22:47:57> on each row, the first term being unix times, the last human time, the middle binary indicating which machine event happened
# open document to read from
l = open('./columns_sorted.txt', 'w')
# CREATE ARRAY CALLED EVENTS
events = collections.deque()
i = 1
# FILL ARRAY WITH "FACTS" ROWS; SPLIT INTO FIELDS, CHANGE TYPES AS APPROPRIATE
for line in j: # columns_unsorted
line = line.rstrip('\n')
string_j = line.split(' ')
time = str(string_j[0])
fan_on = int(string_j[1])
fan_off = int(string_j[2])
heater_on = int(string_j[3])
heater_off = int(string_j[4])
space_on = int(string_j[5])
space_off = int(string_j[6])
pump_on = int(string_j[7])
pump_off = int(string_j[8])
event_time = str(string_j[9])
row = time, fan_on, fan_off, heater_on, heater_off, space_on, space_off, pump_on, pump_off, event_time
events.append(row)
You are missing the readlines function, no?
You have to do:
j = open('./columns_unsorted.txt', 'r')
l = j.readlines()
for line in l:
# what you want to do with each line
In the future, you should print some of your variables, just to be sure the code is working as you want it to, and to help you identifying problems.
(for example, if in your code you would print string_j you would see what kind of problem you have)
Problem was an inconsistent line in the data file. Forgive my haste in posting