Disclaimer: This is not my code. It originated from graphics.py created by Dr. John Zelle.
Where is rate declared as a number? I understand that it is a keyword argument of type None, but how is this pauselength = 1/rate-(now-_update_lasttime) valid? To my knowledge, this is saying that 1 is divisible by None.
import time, os, sys
try: # import as appropriate for 2.x vs. 3.x
import tkinter as tk
except:
import Tkinter as tk
# global variables and functions
_root = tk.Tk()
_root.withdraw()
_update_lasttime = time.time()
def update(rate=None):
global _update_lasttime
if rate:
now = time.time()
pauseLength = 1/rate-(now-_update_lasttime)
if pauseLength > 0:
time.sleep(pauseLength)
_update_lasttime = now + pauseLength
else:
_update_lasttime = now
_root.update()
A simple experiment:
a = None
b = 1
print(b/a)
Reveals the following error (Which makes complete sense):
TypeError: unsupported operand type(s) for /: 'int' and 'NoneType'
If rate is None, the conditional if rate: is false, so none of the code in the following block (including the problematic division) runs. The only code which runs in that situation is _root.update().
Related
My pubsub pull subscription is sending over the message and a None value for each message. I need to find a way to filter out the none values as part of my pipeline processing
Of course some help preventing the none values from arriving from the pull subscription would be nice. But I feel like I'm missing something about the general workflow of defining & applying functions via ParDo.
I've set up a function to filter out none values which seems to work based on a print to console check, however when applying a lambda function that crashes on none types I still receive errors.
I've found the documentation on the python Apache Beam SDK a little sparse but I have been looking all through there for answers without much luck.
from __future__ import absolute_import
import argparse
import logging
from past.builtins import unicode
import apache_beam as beam
import apache_beam.transforms.window as window
from apache_beam.examples.wordcount import WordExtractingDoFn
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.options.pipeline_options import StandardOptions
def print_row(row):
print row
print type(row)
def filter_out_nones(row):
if row is not None:
yield row
else:
print 'we found a none! get it out'
def run(argv=None):
pipeline_options = PipelineOptions()
pipeline_options.view_as(SetupOptions).save_main_session = True
pipeline_options.view_as(StandardOptions).streaming = True
p = beam.Pipeline(options=pipeline_options)
data = ['test1 message','test2 message',None,'test3 please work']
## this does seem to return only the values I would hope for based on the console log
testlogOnly = (p | "makeData" >> beam.Create(data)
| "filter" >> beam.ParDo(filter_out_nones)
| "printtesting" >> beam.Map(print_row))
# | 'encoding' >> beam.Map(lambda x: x.encode('utf-8')).with_output_types(bytes)
# | "writing" >> beam.io.WriteToPubSub("projects/??/topics/??"))
## testlogAndWrite = (p | "MakeWriteData" >> beam.Create(data)
# | "filterHere" >> beam.ParDo(filter_out_nones)
# | "printHere" >> beam.Map(print_row)
## below here does not work due to the following message
## AttributeError: 'NoneType' object has no attribute 'encode' [while running 'encodeHere']
# | 'encodeHere' >> beam.Map(lambda x: x.encode('utf-8')).with_output_types(bytes)
# | "writeTest" >> beam.io.WriteToPubSub("projects/??/topics/??"))
result = p.run()
result.wait_until_finish()
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
run()
If I could log the byte string encoded messages without the none results I'll be where I need to be.
Your approach to filter out None values looks good to me.
However, if I understand it correctly, when you are using testlogAndWrite and get the AttributeError you are keeping the "printHere" >> beam.Map(print_row) step in the pipeline.
print_row reads the messages and prints them but it does not output anything. Therefore, there will be no input for next step encode_here.
To solve this you can comment out that step or make sure that each element is returned:
def print_row(row):
print row
print type(row)
return row
Output:
test1 message
<type 'str'>
test2 message
<type 'str'>
we found a none! get it out
test3 please work
<type 'str'>
I want to use multiprocessing for a project of mine. I have a code that already works fine an i want to adapt it. I realised that when there is a list in the workers method i get the following error; local variable 'RouteInfo' referenced before assignment. Any suggestions?
if __name__ == '__main__':
time_start = time.clock()
AREA_ID, PARENT_ID, routes_data, routingPlanID, StationName = getRouteDetails(StationName, PARENT_ID, AREA_ID)
copyDailyABPDetail(PARENT_ID, AREA_ID)
AREA_ID=AREA_ID.decode('utf-8')
StationsList = [unicode(item[0]) for item in routes_data]
func = partial(testMethod, AREA_ID, PARENT_ID, routingPlanID, StationNameDic)
# declare number of cores to use
cpuNum = multiprocessing.cpu_count()
# Create the pool object
pool = multiprocessing.Pool(processes=cpuNum)
# Fire off list to worker function.
results=pool.map(func,StationsList)
pool.close()
pool.join()
print results
time_elapsed = (time.clock() - time_start)
def testMethod(AREA_ID, PARENT_ID, routingPlanID, StationNameDic, StationName):
try:
print "Processing: ",StationName
RouteInfo = ["ALL", "ALL", "PRESERVE_BOTH", ["OnewaySelective","Footway","Steps","Path","Tolls"], "OR", "Α,Β,Γ", "NGIS", "RA"]
print RouteInfo
except Exception as e:
# If an error occurred, print line number and error message
import traceback, sys
tb = sys.exc_info()[2]
print "An error occured on line %i" % tb.tb_lineno
print str(e)
return StationName, RouteInfo
Solved there was an error in my code (variable StatioName had an utf-8 issue), so the variable RouteInfo was never been assigned.
I was reading a similar question Returning error string from a function in python. While I experimenting to create something similar in an Object Oriented programming so I could learn a few more things I got lost.
I am using Python 2.7 and I am a beginner on Object Oriented programming.
I can not figure out how to make it work.
Sample code checkArgumentInput.py:
#!/usr/bin/python
__author__ = 'author'
class Error(Exception):
"""Base class for exceptions in this module."""
pass
class ArgumentValidationError(Error):
pass
def __init__(self, arguments):
self.arguments = arguments
def print_method(self, input_arguments):
if len(input_arguments) != 3:
raise ArgumentValidationError("Error on argument input!")
else:
self.arguments = input_arguments
return self.arguments
And on the main.py script:
#!/usr/bin/python
import checkArgumentInput
__author__ = 'author'
argsValidation = checkArgumentInput.ArgumentValidationError(sys.argv)
if __name__ == '__main__':
try:
result = argsValidation.validate_argument_input(sys.argv)
print result
except checkArgumentInput.ArgumentValidationError as exception:
# handle exception here and get error message
print exception.message
When I am executing the main.py script it produces two blank lines. Even if I do not provide any arguments as input or even if I do provide argument(s) input.
So my question is how to make it work?
I know that there is a module that can do that work for me, by checking argument input argparse but I want to implement something that I could use in other cases also (try, except).
Thank you in advance for the time and effort reading and replying to my question.
OK. So, usually the function sys.argv[] is called with brackets in the end of it, and with a number between the brackets, like: sys.argv[1]. This function will read your command line input. Exp.: sys.argv[0] is the name of the file.
main.py 42
In this case main.py is sys.argv[0] and 42 is sys.argv[1].
You need to identifi the string you're gonna take from the command line.
I think that this is the problem.
For more info: https://docs.python.org/2/library/sys.html
I made some research and I found this useful question/ answer that helped me out to understand my error: Manually raising (throwing) an exception in Python
I am posting the correct functional code under, just in case that someone will benefit in future.
Sample code checkArgumentInput.py:
#!/usr/bin/python
__author__ = 'author'
class ArgumentLookupError(LookupError):
pass
def __init__(self, *args): # *args because I do not know the number of args (input from terminal)
self.output = None
self.argument_list = args
def validate_argument_input(self, argument_input_list):
if len(argument_input_list) != 3:
raise ValueError('Error on argument input!')
else:
self.output = "Success"
return self.output
The second part main.py:
#!/usr/bin/python
import sys
import checkArgumentInput
__author__ = 'author'
argsValidation = checkArgumentInput.ArgumentLookupError(sys.argv)
if __name__ == '__main__':
try:
result = argsValidation.validate_argument_input(sys.argv)
print result
except ValueError as exception:
# handle exception here and get error message
print exception.message
The following code prints: Error on argument input! as expected, because I violating the condition.
Any way thank you all for your time and effort, hope this answer will help someone else in future.
I have been trying to get over my fear of Cython (fear because I literally know NOTHING about c, or c++)
I have a function which takes 2 arguments, a set (we'll call it testSet), and a list of sets (we'll call that targetSets). The function then iterates through targetSets, and computes the length of the intersection with testSet, adding that value to a list, which is then returned.
Now, this isn't by itself that slow, but the problem is I need to do simulations of the testSet (and a large number at that, ~ 10,000), and the targetSet is about 10,000 sets long.
So for a small number of simulations to test, the pure python implementation was taking ~50 secs.
I tried making a cython function, and it worked and it's now running at ~16 secs.
If there is anything else that I could do to the cython function that anyone could think of that would be great (python 2.7 btw)
Here is my Cython implementation in overlapFunc.pyx
def computeOverlap(set testSet, list targetSets):
cdef list obsOverlaps = []
cdef int i, N
cdef set overlap
N = len(targetSets)
for i in range(N):
overlap = testSet & targetSets[i]
if len(overlap) <= 1:
obsOverlaps.append(0)
else:
obsOverlaps.append(len(overlap))
return obsOverlaps
and the setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [Extension("overlapFunc",
["overlapFunc.pyx"])]
setup(
name = 'computeOverlap function',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
and some code to build some random sets for testing and to time the function. test.py
import numpy as np
from overlapFunc import computeOverlap
import time
def simRandomSet(n):
for i in range(n):
simSet= set(np.random.randint(low=1, high=100, size=50))
yield simSet
if __name__ == '__main__':
np.random.seed(23032014)
targetSet = [set(np.random.randint(low=1, high=100, size=50)) for i in range(10000)]
simulatedTestSets = simRandomSet(200)
start = time.time()
for i in simulatedTestSets:
obsOverlaps = computeOverlap(i, targetSet)
print time.time()-start
I tried changing the def at the start of the computerOverlap function, as in:
cdef list computeOverlap(set testSet, list targetSets):
but I get the following warning message when I run the setup.py script:
'__pyx_f_11overlapFunc_computeOverlap' defined but not used [-Wunused-function]
and then when I run something that tries to use the function I get an import Error:
from overlapFunc import computeOverlap
ImportError: cannot import name computeOverlap
Thanks in advance for your help,
Cheers,
Davy
In the following line, the extension module name and the filename does not match actual filename.
ext_modules = [Extension("computeOverlapWithGeneList",
["computeOverlapWithGeneList.pyx"])]
Replace it with:
ext_modules = [Extension("overlapFunc",
["overlapFunc.pyx"])]
I'm trying to use the Python package ComplexNetworkSim, which inherits from networkx and SimPy, to simulate an agent-based model of how messages propagate within networks.
Here is my code:
from ComplexNetworkSim import NetworkSimulation, NetworkAgent, Sim
import networkx as nx
#define constants for our example of states
NO_MESSAGE = 0
MESSAGE = 1
class Message(object):
def __init__(self,topic_pref):
self.relevance = topic_pref
class myAgent(NetworkAgent):
def __init__(self, state, initialiser):
NetworkAgent.__init__(self, state, initialiser)
self.state = MESSAGE
self.topic_pref = 0.5
def Run(self):
while True:
if self.state == MESSAGE:
self.message = self.Message(topic_pref, self, TIMESTEP)
yield Sim.hold, self, NetworkAgent.TIMESTEP_DEFAULT
elif self.state == NO_MESSAGE:
yield Sim.hold, self, NetworkAgent.TIMESTEP_DEFAULT
# Network and initial states of agents
nodes = 30
G = nx.scale_free_graph(nodes)
states = [MESSAGE for n in G.nodes()]
# Simulation constants
MAX_SIMULATION_TIME = 25.0
TRIALS = 2
def main():
directory = 'test' #output directory
# run simulation with parameters
# - complex network structure
# - initial state list
# - agent behaviour class
# - output directory
# - maximum simulation time
# - number of trials
simulation = NetworkSimulation(G,
states,
myAgent,
directory,
MAX_SIMULATION_TIME,
TRIALS)
simulation.runSimulation()
if __name__ == '__main__':
main()
(There may be other problems downstream with this code and it is not fully tested.)
My problem is that the myAgent object is not properly calling the method Run as an attribute. Specifically, this is the error message that I get when I try to run the above code:
Starting simulations...
---Trial 0 ---
set up agents...
Traceback (most recent call last):
File "simmessage.py", line 55, in <module>
main()
File "simmessage.py", line 52, in main
simulation.runSimulation()
File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages/ComplexNetworkSim-0.1.2-py2.7.egg/ComplexNetworkSim/simulation.py", line 71, in runSimulation
self.runTrial(i)
File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/site-packages/ComplexNetworkSim-0.1.2-py2.7.egg/ComplexNetworkSim/simulation.py", line 88, in runTrial
self.activate(agent, agent.Run())
AttributeError: 'myAgent' object has no attribute 'Run'
Does anybody know why this is? I can't figure how my code differs substantially from the example in ComplexNetworkSim.
I've run your code on my machine and there the Run method gets called.
My best guess is what Paulo Scardine wrote, but since i can't reproduce the problem i can't actually debug it.