I would like to write the profilehooks output to a .prof file. In order to convert the file to a qcachgrind file and visualize the profiling results. (I can't use cProfile as cProfile, does not profile the code)
Its not on the web, but the documentation says:
profile(fn=None, skip=0, filename=None, immediate=False, dirs=False, sort=None, entries=40, profiler=('cProfile', 'profile', 'hotshot'), stdout=True)
Mark `fn` for profiling.
If `skip` is > 0, first `skip` calls to `fn` will not be profiled.
If `immediate` is False, profiling results will be printed to
sys.stdout on program termination. Otherwise results will be printed
after each call. (If you don't want this, set stdout=False and specify a
`filename` to store profile data.)
If `dirs` is False only the name of the file will be printed.
Otherwise the full path is used.
`sort` can be a list of sort keys (defaulting to ['cumulative',
'time', 'calls']). The following ones are recognized::
'calls' -- call count
'cumulative' -- cumulative time
'file' -- file name
'line' -- line number
'module' -- file name
'name' -- function name
'nfl' -- name/file/line
'pcalls' -- call count
'stdname' -- standard name
'time' -- internal time
`entries` limits the output to the first N entries.
`profiler` can be used to select the preferred profiler, or specify a
sequence of them, in order of preference. The default is ('cProfile'.
'profile', 'hotshot').
If `filename` is specified, the profile stats will be stored in the
named file. You can load them with pstats.Stats(filename) or use a
visualization tool like RunSnakeRun.
Usage::
def fn(...):
...
fn = profile(fn, skip=1)
If you are using Python 2.4, you should be able to use the decorator
syntax::
#profile(skip=3)
def fn(...):
...
or just ::
#profile
def fn(...):
...
Related
While searching a product_set using command line, we can set a parameter features.max Results to limit the number of items returned in the response.
Field-specific considerations:
features.maxResults - The maximum number of results to be returned.
But how can I pass this parameter while running the python script provided here?
Also, does this parameter affect the response time?
The documentation is really not very well written, but I found that you just have to add the parameter max_results=<max> inside the method product_search as following:
response = image_annotator_client.product_search(
image, image_context=image_context,max_results=2)
To confirm that it works, I've added the an debug line inside the file on the google library /usr/local/lib/python3.7/dist-packages/google/cloud/vision_helpers/decorators.py
And when I run the code, I get the printout of the request made to the api, and as you can see, the parameter is correctly defined:
# python3 search.py
Result:
#BEGIN DEBUG:
{'image': source {
image_uri: "gs://<bucket>/short-blue.jpeg"
}
, 'features': [{'type_': <Type.PRODUCT_SEARCH: 12>, 'max_results': 2}], 'image_context': product_search_params {
product_set: "projects/<project>/locations/us-west1/productSets/product-set-name"
product_categories: "apparel-v2"
filter: "color:blue"
}
}
#END DEBUG:
Product set index time:
None
Search results:
I am experimenting using elasticsearch in a dummy project in django. I am attempting to make a search page using django-elasticsearch-dsl. The user may provide a title, summary and a score to search for. The search should match all the information given by the user, but if the user does not provide any info about something, this should be skipped.
I am running the following code to search for all the values.
client = Elasticsearch()
s = Search().using(client).query("match", title=title_value)\
.query("match", summary=summary_value)\
.filter('range', score={'gt': scorefrom_value, 'lte': scoreto_value})
When I have a value for all the fields then the search works correctly, but if for example I do not provide a value for the summary_value, although I am expecting the search to continue searching for the rest of the values, the result is that it comes up with nothing as a result.
Is there some value that the fields should have by default in case the user does not provide a value? Or how should I approach this?
UPDATE 1
I tried using the following, but it returns every time no matter the input i am giving the same results.
s = Search(using=client)
if title:
s.query("match", title=title)
if summary:
s.query("match", summary=summary)
response = s.execute()
UPDATE 2
I can print using the to_dict().
if it is like the following then s is empty
s = Search(using=client)
s.query("match", title=title)
if it is like this
s = Search(using=client).query("match", title=title)
then it works properly but still if i add s.query("match", summary=summary) it does nothing.
You need to assign back into s:
if title:
s = s.query("match", title=title)
if summary:
s = s.query("match", summary=summary)
I can see in the Search example that django-elasticsearch-dsl lets you apply aggregations after a search so...
How about "staging" your search? I can think if the following:
#first, declare the Search object
s = Search(using=client, index="my-index")
#if parameter1 exists
if parameter1:
s.filter("term", field1= parameter1)
#if parameter2 exists
if parameter2:
s.query("match", field=parameter2)
Do the same for all your parameters (with the needed method for each) so only the ones that exist will appear in your query. At the end just run
response = s.execute()
and everything should work as you want :D
I would recommend you to use the Python ES Client. It lets you manage multiple things related to your cluster: set mappings, health checks, do queries, etc.
In its method .search(), the body parameter is where you send your query as you normally would run it ({"query"...}). Check the Usage example.
Now, for your particular case, you can have a template of your query stored in a variable. You first start with, let's say, an "empty query" only with filter, just like:
query = {
"query":{
"bool":{
"filter":[
]
}
}
}
From here, you now can build your query from the parameters you have.
This is:
#This would look a little messy, but it's useful ;)
#if parameter1 is not None or emtpy
#(change the if statement for your particular case)
if parameter1:
query["query"]["bool"]["filter"].append({"term": {"field1": parameter1}})
Do the same for all your parameters (for strings, use "term", for ranges use "range" as usual) and send the query in the .search()'s body parameter and it should work as you want.
Hope this is helpful! :D
I am trying to use ConfigParser in the following situation. I am running some code after which i have an object with several attributes. I pick out some of these attributes and write them to a .ini file with configparser. This works fine and my .ini file looks something like this.
[Section]
path = "D:\\"
number = 10.0
boolean = False
Then with some other script I want to read the file and add the items as attributes to another object (self) using.
parser.read('file.ini')
self.__dict__.update(dict(parser.items("Section")))
This fails because all values are read as strings by configparser and now all attributes are strings. parser.items("Section") looks like this:
[('path', '"D:\\"'), ('number', '10.0'), ('boolean', 'False')]
Now I could go and specify the floats, integers, and booleans by their keys and use the corresponding methods parser.getfloat, getint, and getboolean to get the right python types out. However, that means making an extra list of keys and a loop to get them for each data type, which i don't want. Furthermore, even the path is now double quoted and i need to start removing quotes.
This behavior makes ConfigParser almost completely worthless to me and I am doubting if I am using it correctly an if ConfigParser is the best option for my goal, which is simply to write object attributes to a file and at a later time read the file and add the parameters to a new object as attributes. Can I use ConfigParser for this effectively? Or should I start looking for a different method?
INI is not JSON. There are no data types. There are sections and key/value pairs. Stuff before the = is the key, stuff after it is the value. Everything is text.
There are no "strings", which means there is no need to double quote anything. The same goes for "escaped" backslashes. The concept of escaping does not exist in the INI file format.
So, first off, your file should be looking like this:
[Section]
path = D:\
number = 10.0
boolean = False
Next, I consider this a dangerous operation
parser.read('file.ini')
self.__dict__.update(dict(parser.items("Section")))
because it potentially trashes your class instance with arbitrary values from a text file that might contain garbage, but when you can swear that the file will always be fine, well, do it if you must.
I would write a more explicit way of reading and validating config data. I sense your reason not to do that is laziness or a false sense of simplicity; both of these reasons are not very good.
If you want a semi-automated way of type conversion, tag the keys with a data type.
[Section]
s_path = D:\
f_number = 10.0
b_boolean = False
and write a function that knows how to handle them and throws when something is not right.
def type_convert(items):
result = []
for (key, value) in items:
type_tag = key[:2]
if type_tag == "s_":
result.append((key[2:], value))
elif type_tag == "f_":
result.append((key[2:], float(value)))
elif type_tag == "b_":
result.append((key[2:], bool(value)))
else:
raise ValueError('Invalid type tag "%s" found in ini file.' % type_tag)
# alternatively: "everything else defaults to string"
return result
which you can use to make the operation somewhat more convenient:
self.__dict__.update(dict(type_convert(parser.items("Section"))))
Of course you still run the risk of trashing your class instance by overriding keys that should not be overridden.
I have a list including 4000 elements in python which each of its elements is an object of following class with several values.
class Point:
def __init__(self):
self.coords = []
self.IP=[]
self.BW=20
self.status='M'
def __repr__(self):
return str(self.coords)
I do not know how to save this list for future uses.
I have tried to save it by open a file and write() function, but this is not what I want.
I want to save it and import it in next program, like what we do in MATLAB that we can save a variable and import it in future
pickle is a good choice:
import pickle
with open("output.bin", "wb") as output:
pickle.dump(yourList, output)
and symmetric:
import pickle
with open("output.bin", "rb") as data:
yourList = pickle.load(data)
It is a good choice because it is included with the standard library, it can serialize almost any Python object without effort and has a good implementation, although the output is not human readable. Please note that you should use pickle only for your personal scripts, since it will happily load anything it receives, including malicious code: I would not recommend it for production or released projects.
This might be an option:
f = open('foo', 'wb')
np.save(f, my_list)
for loading then use
data = np.load(open('foo'))
if 'b' is not present in 'wb' then the program gives an error:
TypeError: write() argument must be str, not bytes
"b" for binary makes the difference.
Since you say Matlab, numpy should be an option.
f = open('foo', 'w')
np.save(f, my_list)
# later
data = np.load(open('foo'))
Of course, it'll return an array, not a list, but you can coerce it if you really want an array...
I have created a python script that runs from an ArcMap 10.1 session; however, I would like to modify it to run as a stand alone script, if possible. The problem is I don't see a workaround for prompting the user for the parameters when executed outside ArcMap.
Can this even be reasonably done? If so, how would I approach it? Below is a sample of my script. How can I modify this to prompt the user at the command line for the path names of parameters 0 and 1?
import arcpy
arcpy.env.overwriteOutput = True
siteArea = arcpy.GetParameterAsText(0)
tempGDB_Dir = arcpy.GetParameterAsText(1)
tempGDB = tempGDB_Dir + "\\tempGDB.gdb"
# Data from which records will be extracted
redWoods = "D:\\Data\\GIS\\Landforms\\Tress.gdb\\Redwoods"
# List of tree names that will be used in join
treesOfInterest = "C:\\Data\\GIS\\Trees\\RedwoodList.dbf"
inFeature = [redWoods, siteArea]
tempFC = tempGDB_Dir + "\\TempFC"
tempFC_Layer = "TempFC_Layer"
output_dbf = tempGDB_Dir + "\\Output.dbf"
# Make a temporaty geodatabase
arcpy.CreateFileGDB_management(tempGDB_Dir, "tempGDB.gdb")
# Intersect trees with site area
arcpy.Intersect_analysis([redWoods, siteArea], tempFC, "ALL", "", "INPUT")
# Make a temporary feature layer of the results
arcpy.MakeFeatureLayer_management(tempFC, tempFC_Layer)
# Join redwoods data layer to list of trees
arcpy.AddJoin_management(tempFC_Layer, "TreeID", treesOfInterest, "TreeID", "KEEP_COMMON")
# Frequency analysis - keeps only distinct species values
arcpy.Frequency_analysis(tempFC_Layer, output_dbf, "tempFC.TreeID;tempFC.TreeID", "")
# Delete temporary files
arcpy.Delete_management(tempFC_Layer)
arcpy.Delete_management(tempGDB)
This is as much a philosophical question as it is a programmatic one. I am interested in whether this can be done and the amount of effort to do it this way. Is the effort worth the convenience of not opening up a map document?
Check to see if the parameters were specified. If they were not specified, do one of the following:
Use Python's raw_input() method to prompt the user (see this question).
Print a "usage" message, which instructs the user to enter parameters on the command line, and then exit.
Prompting the user could look like this:
siteArea = arcpy.GetParameterAsText(0)
tempGDB_Dir = arcpy.GetParameterAsText(1)
if (not siteArea):
arcpy.AddMessage("Enter the site area:")
siteArea = raw_input()
if (not tempGDB_Dir):
arcpy.AddMessage("Enter the temp GDB dir:")
tempGDB_Dir = raw_input()
Printing a usage message could look like this:
siteArea = arcpy.GetParameterAsText(0)
tempGDB_Dir = arcpy.GetParameterAsText(1)
if (not (siteArea and tempGDB_Dir)):
arcpy.AddMessage("Usage: myscript.py <site area> <temp GDB dir>")
else:
# the rest of your script goes here
If you prompt for input with raw_input(), make sure to make all parameters required when adding to your toolbox in ArcGIS for Desktop. Otherwise, you'll get this error from raw_input() when running in Desktop:
EOFError: EOF when reading a line
hell yeah its worth the convenience of not opening up arcmap. I like to use the optparse module to create command line tools. arcpy.GetParameter(0) is only useful for Esri GUI integration (e.g. script tools). Here is a nice example of a python commandline tool:
http://www.jperla.com/blog/post/a-clean-python-shell-script
I include a unittest class in my tools to testing and automation. I also keep all arcpy.GetParameterAsText statements outside of any real business logic. I like to include at the bottom:
if __name__ == '__main__':
if arcpy.GetParameterAsText(0):
params = parse_arcpy_parameters()
main_business_logic(params)
else:
unittest.main()