Im trying to write a unit test for 'Summary' but not sure what variables I need to check?
from prometheus_client import Counter, Summary
import unittest
import time
from prometheus_client import REGISTRY
my_summary = Summary('my_summary', 'A useful help string.')
def my_function():
time.sleep(1)
my_summary.observe(5)
class TestMyFunction(unittest.TestCase):
def test_metric_incremented(self):
print 'here'
before = REGISTRY.get_sample_value('my_summary')
print 'summary before == ', before
my_function()
after = REGISTRY.get_sample_value('my_summary')
print 'summary after == ', after
self.assertEqual(0, after - before)
if __name__ == '__main__':
unittest.main()
Here is my code I observe the function 5 seconds. Not sure if this is the right approach..Any test example would be great.
I tried following this blog - https://www.robustperception.io/how-to-unit-test-prometheus-instrumentation/
The time series you want are my_summary_count and my_summary_sum.
Related
I have the following module that I am trying to write unit tests for.
import myModuleWithCtxMgr
def myFunc(arg1):
with myModuleWithCtxMgr.ctxMgr() as ctxMgr:
result = ctxMgr.someFunc()
if result:
return True, result
return False, None
The unit tests I'm working on looks like this.
import mock
import unittest
import myModule as myModule
class MyUnitTests(unittest.TestCase):
#mock.patch("myModuleWithCtxMgr.ctxMgr")
def testMyFunc(self, mockFunc):
mockReturn = mock.MagicMock()
mockReturn.someFunc = mock.Mock(return_value="val")
mockFunc.return_value = mockReturn
result = myModule.myFunc("arg")
self.assertEqual(result, (True, "val"))
The test is failing because result[0] = magicMock() and not the return value (I thought) I configured.
I've tried a few different variations of the test but I can't seem to be able to mock the return value of ctxMgr.someFunc(). Does anyone know how I might accomplish this?
Thanks!
The error says:
First differing element 1:
<MagicMock name='ctxMgr().__enter__().someFunc()' id='139943278730000'>
'val'
- (True, <MagicMock name='ctxMgr().__enter__().someFunc()' id='139943278730000'>)
+ (True, 'val')
The error contains the mock name which exactly shows you what needs to be mocked. Note that __enter__ corresponds to the Context Manager protocol.
This works for me:
class MyUnitTests(unittest.TestCase):
#mock.patch("myModuleWithCtxMgr.ctxMgr")
def testMyFunc(self, mockCtxMgr):
mockCtxMgr().__enter__().someFunc.return_value = "val"
result = myModule.myFunc("arg")
self.assertEqual(result, (True, "val"))
Note how each of these is a separate MagicMock instance which you can configure:
mockCtxMgr
mockCtxMgr()
mockCtxMgr().__enter__
mockCtxMgr().__enter__()
mockCtxMgr().__enter__().someFunc
MagicMocks are created lazily but have identity, so you can configure them this way and it Just Works.
I'm trying to read a table, modify a column and write to another table. I followed the available documentation and ran following code. It doesn't give any errors, but the task doesn't get performed either.
I tried removing the transformation step and then information gets written.
import sqlalchemy
import bonobo
import bonobo_sqlalchemy
def get_services():
return {
'sql_alchemy.engine': sqlalchemy.create_engine('postgresql://postgres:password#localhost:5432/postgres')
}
def transform(*row):
new_row = row[0]+1, row[1]
yield new_row
def get_graph(**options):
graph = bonobo.Graph()
graph.add_chain(bonobo_sqlalchemy.Select('SELECT * FROM users', engine='sql_alchemy.engine')
,
transform,
bonobo_sqlalchemy.InsertOrUpdate(table_name='table_1', engine='sql_alchemy.engine'),
)
return graph
# The __main__ block actually execute the graph.
if __name__ == '__main__':
parser = bonobo.get_argument_parser()
with bonobo.parse_args(parser) as options:
bonobo.run(get_graph(**options), services=get_services(**options))
Output:
- Select in=1 out=6 [done]
- format_for_db in=6 out=6 [done]
- InsertOrUpdate in=6 out=6 [done]
It works when a Dictionary is yielded as follows,
yield {"id": row[0], "text": row[1], "count":row[2]}
with bonobo.UnpackItems(0) node in the chain after the transformation.
I need some help regarding the error in code. My Code consists of retrieving the zomato reviews and storing it in HDFS and again reading it performing Recommender Analtyics on it. I am getting a problem regarding my function is not recognizing in pyspark code. I am not entirely pasting the code as it might be confusing so i am writing a small similar use case for your easy understanding.
I am trying to read a file from local and converting it to dataframe from rdd and performing some operations and again converting it to rdd and performing map operation to have delimiter by '|' and then save it to HDFS.
When i try to call self.filter_data(y) in lambda func of check function its not recognizing and giving me error as
Exception: It appears that you are attempting to reference
SparkContext from a broadcast variable, action, or transformation.
SparkContext can only be used on the driver, not in code that it run
on workers. For more information, see SPARK-5063.
****CAN ANY ONE HELP ME WHY MY FILTER_DATA FUNCTION IS NOT RECOGNISING? SHOULD I NEED TO ADD ANY THING OR ANY THING WRONG IN THE WAY I AM CALLING. PLEASE HELP ME. THANKS IN ADVANCE****
INPUT VALUE
starting
0|0|ffae4f|0|https://b.zmtcdn.com/data/user_profile_pictures/565/aed32fa2eb18bb4a5a3ba426870fd565.jpg?fit=around%7C100%3A100&crop=100%3A100%3B%2A%2C%2A|https://www.zomato.com/akellaram87?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1|2.5|FFBA00|Well...|unknown|16946626|2017-08-01T00-25-43.455182Z|30059877|Have been here for a quick bite for lunch, ambience and everything looked good, food was okay but presentation was not very appealing. We or...|2017-04-15 16:38:38|Big Foodie|6|Venkata Ram Akella|akellaram87|Bad Food|0.969352505662|0|0|0|0|0|0|1|1|0|0|1|0|0|0.782388212399
ending
starting
1|0|ffae4f|0|https://b.zmtcdn.com/data/user_profile_pictures/4d1/d70d7a57e1bfdf296ff4db3d8daf94d1.jpg?fit=around%7C100%3A100&crop=100%3A100%3B%2A%2C%2A|https://www.zomato.com/users/sm4-2011696?utm_source=api_basic_user&utm_medium=api&utm_campaign=v2.1|1|CB202D|Avoid!|unknown|16946626|2017-08-01T00-25-43.455182Z|29123338|Giving a 1.0 rating because one cannot proceed with writing a review, without rating it. This restaurant deserves a 0 star rating. The qual...|2017-01-04 10:54:53|Big Foodie|4|Sm4|unknown|Bad Service|0.964402034541|0|1|0|0|0|0|0|1|0|0|0|1|0|0.814540622345
ending
My code:
if __name__== '__main__':
import os,logging,sys,time,pandas,json;from subprocess
import PIPE,Popen,call;from datetime import datetime, time, timedelta
from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName('test')
sc = SparkContext(conf = conf,pyFiles=['/bdaas/exe/nlu_project/spark_classifier.py','/bdaas/exe/spark_zomato/other_files/spark_zipcode.py','/bdaas/exe/spark_zomato/other_files/spark_zomato.py','/bdaas/exe/spark_zomato/conf_files/spark_conf.py','/bdaas/exe/spark_zomato/conf_files/date_comparision.py'])
from pyspark.sql import Row, SQLContext,HiveContext
from pyspark.sql.functions import lit
sqlContext = HiveContext(sc)
import sys,logging,pandas as pd
import spark_conf
n = new()
n.check()
class new:
def __init__(self):
print 'entered into init'
def check(self):
data = sc.textFile('file:///bdaas/src/spark_dependencies/classifier_data/final_Output.txt').map(lambda x: x.split('|')).map(lambda z: Row(restaurant_id=z[0], rating = z[1], review_id = z[2],review_text = z[3],rating_color = z[4],rating_time_friendly=z[5],rating_text=z[6],time_stamp=z[7],likes=z[8],comment_count =z[9],user_name = z[10],user_zomatohandle=z[11],user_foodie_level = z[12],user_level_num=z[13],foodie_color=z[14],profile_url=z[15],profile_image=z[16],retrieved_time=z[17]))
data_r = sqlContext.createDataFrame(data)
data_r.show()
d = data_r.rdd.collect()
print d
data_r.rdd.map(lambda x: list(x)).map(lambda y: self.filter_data(y)).collect()
print data_r
def filter_data(self,y):
s = str()
for i in y:
print i.encode('utf-8')
if i != '':
s = s + i.encode('utf-8') + '|'
print s[0:-1]
return s[0:-1]
I am using multiprocessing pool in one of my program and i need the program to have infinite loop which means run forever... But after i applied the while True: loop in the pool program, the program seem like forever hang and do nothing at there...
I come out with a very simple Python code as shown as below:
from multiprocessing import Pool
def market_update(var):
return var*var
while True:
alist = [1,2,3,4,5,6,7,8,9,10,11,12]
print alist
if __name__ == '__main__':
pool = Pool()
result= pool.map(market_update, alist)
print result
Can anyone explain why and if there any wokraround?
Edited code as suggested from Dano:
from multiprocessing import Pool
def market_update(var):
return var*var
if __name__ == '__main__':
pool = Pool()
while True:
alist = [1,2,3,4,5,6,7,8,9,10,11,12]
print alist
result = pool.map(market_update, alist)
print result
I got a function in a certain module that I want to redefine(mock) at runtime for testing purposes. As far as I understand, function definition is nothing more than an assignment in python(the module definition itself is a kind of function being executed). As I said, I wanna do this in the setup of a test case, so the function to be redefined lives in another module. What is the syntax for doing this?
For example, 'module1' is my module and 'func1' is my function, in my testcase I have tried this (no success):
import module1
module1.func1 = lambda x: return True
import module1
import unittest
class MyTest(unittest.TestCase):
def setUp(self):
# Replace othermod.function with our own mock
self.old_func1 = module1.func1
module1.func1 = self.my_new_func1
def tearDown(self):
module1.func1 = self.old_func1
def my_new_func1(self, x):
"""A mock othermod.function just for our tests."""
return True
def test_func1(self):
module1.func1("arg1")
Lots of mocking libraries provide tools for doing this sort of mocking, you should investigate them as you will likely get a good deal of help from them.
import foo
def bar(x):
pass
foo.bar = bar
Just assign a new function or lambda to the old name:
>>> def f(x):
... return x+1
...
>>> f(3)
4
>>> def new_f(x):
... return x-1
...
>>> f = new_f
>>> f(3)
2
It works also when a function is from another module:
### In other.py:
# def f(x):
# return x+1
###
import other
other.f = lambda x: x-1
print other.f(1) # prints 0, not 2
Use redef: http://github.com/joeheyming/redef
import module1
from redef import redef
rd_f1 = redef(module1, 'func1', lambda x: True)
When rd_f1 goes out of scope or is deleted, func1 will go back to being back to normal
If you want to reload into the interpreter file foo.py that you are editing, you can make a simple-to-type function and use execfile(), but I just learned that it doesn't work without the global list of all functions (sadly), unless someone has a better idea:
Somewhere in file foo.py:
def refoo ():
global fooFun1, fooFun2
execfile("foo.py")
In the python interpreter:
refoo() # You now have your latest edits from foo.py