Is is possible in Python to have a generator that yields values in a loop be alternatively called as a normal function where the final value of that loop is returned? I tried setting a flag as argument and then choose to yield or return depending on that flag. But the mere existence of the keyword yield in a function transforms it automatically into a generator and Python complains that there was a return statement in a generator.
Here an example of such a function:
def function(generator=True):
a = 0
for i in range(10):
a = i
if generator:
yield a
if not generator:
return a
Such a function would be useful for me when in some cases I just need the final result (eg. using it as residual function for optimization) while in other cases I need the incremental results after each iteration (for example using a differential model for a robot, updating the robot's pose with each new velocity command). For now I am having two functions where one has the yield and the other has the return. So is it possible to combine those two?
You'll still have to iterate over it to get its single value, but you can use yield again instead of return to accomplish this.
>>> def function(generator=True):
... a = 0
... for i in range(10):
... a = i
... if generator:
... yield a
... if not generator:
... yield a
...
>>> a = function()
>>> print(*a)
0 1 2 3 4 5 6 7 8 9
>>> a = function(0)
>>> print(a)
<generator object function at 0x0000000001603240>
>>> print(*a)
9
>>> a = function(0)
Note that having return inside a generator is a SyntaxError in Python 2, but not Python 3. Replacing the return with yield produces the same result in 2 and 3.
it's still a generator, even calling return.
I'd to not mix both generator/regular function anyway.
You can wrap something on top of the iterator, in case you need to loop through the results anyway.
A simple code that could do what you want:
last = None
for last in function(): pass
Now last holds the value you want and you can use it on your code.
The return inside a generator was added to Python 3.3
And it's equivalent to StopIteration(value)
return expr in a generator causes StopIteration(expr) to be raised
upon exit from the generator.
In a generator, the statement
return value is semantically equivalent to
raise StopIteration(value) except that, as currently, the exception
cannot be caught by except clauses within the returning generator.
Related
I'm new to Keras framework and I want to implement the following loss function of
Root Mean Squared Logarithmic Error
Here is my code for Keras with tensorflow backend
def loss_function(y_true, y_pred):
ones = K.ones(shape=K.shape(y_pred).shape)
y_pred = tf.add(y_pred,ones)
y_true = tf.add(y_true,ones)
val = K.sqrt(K.mean(K.sum(K.log(y_pred)-K.log(y_true))))
return val
But I end up getting the following error:
ValueError: Error when checking input: expected dense_1_input to have shape (None, 16) but got array with shape (1312779, 11)
with the val returned to be 0.
The order of your operations is inverted.
Since "log(true) - log(pred)" can be either negative or positive (the result may be a little higher or a little lower than the expected), the square is the first thing that must happen. (The square is responsible for eliminating the negative signs).
And the mean is the last one (the most external), because you want first to compute the error for each element, and only after that you get the mean of the error. (The mean function already carries the sum function in it).
So:
def loss_function(y_true, y_pred):
y_pred = y_pred + 1
y_true = y_true + 1
return K.mean(K.square(K.log(y_pred)-K.log(y_true)))
Please note that this does not carry the "root" part. If you want to add it, I'd say that the root should go before the mean (different from the formula in the picture)
I'd use this instead:
return K.mean(K.sqrt(K.square(K.log(y_pred)-K.log(y_true))))
Make sure that your model ends with an activation that outputs numbers greater or equal to zero:
Relu is ok
Sigmoid is ok
Softmax is ok
Other activations may have negative values and will bring errors with log:
linear is not ok
tanh is not ok
lang_group = 'en'
for place_category in place['categories']:
translation, created = \
PlaceTypesTranslations.objects.get_or_create(
name=place_category, lang_group=lang_group,
defaults={'place_type_group': PlaceTypesGroups.objects.create()})
In this case if the loop has 1000 iterations and, for example, 500 times created=True and other 500 times created=False, anyway will be created 1000 PlaceTypesGroups, so for some reason even if get_or_create returns get, defaults anyway is executed.
The same algorithm, but different approach:
lang_group = 'en'
for place_category in place['categories']:
if PlaceTypesTranslations.objects.filter(name=place_category, lang_group=lang_group).exists():
place_cat_trans = PlaceTypesTranslations.objects.get(name=place_category, lang_group=lang_group)
place_group = place_cat_trans.place_type_group
else:
place_group = PlaceTypesGroups.objects.create()
place_cat_trans = PlaceTypesTranslations.objects.create(name=place_category,
lang_group=lang_group,
place_type_group=place_group)
In this case just 500 times will be created PlaceTypesGroups as expected.
Why is that? What I do not see in the 1st case? Why does get_or_create creates 1000 PlaceTypesGroups?
That's just the way Python expressions always work. Anything inside an expression must always be fully evaluated before the expression itself can be passed to a function or method.
However, Django specifically lets you pass a callable, rather than a value, in the defaults hash. So you can do:
PlaceTypesTranslations.objects.get_or_create(
name=place_category, lang_group=lang_group,
defaults={'place_type_group': PlaceTypesGroups.objects.create})
and it will call the create method as required.
it's called 1000x, because you are assigning the returned value from the function. I'll start with a simple example:
place_type_group = some_function()
The variable now contains whatever the function returns, right?
Now if you wrap it into a dictionary, it's still the same thing, just wrapper into a dictionary:
dict(place_type_group = some_function())
the element in a dict still contains the value returned from some_function()
The dictionary above is just equal for the following code, which is what you do in your code (ie. assigning function return value into a variable)
{'place_type_group': some_function() }
In Python 2 iterators offer .next(), a callable method:
it = iter(xrange(10))
it.next()
> 0
it.next()
> 1
...
In Python 3 one has to use the built-in function next():
it = iter(range(10))
next(it)
> 0
next(it)
> 1
...
Is this just "syntactic sugar"? Like making it more obvious to use next() by moving it into the built-in functions? Or does any advanced concept hide behind this change?
You are directly asking about PEP 3114
consider the following code:
class test:
def __iter__(self):
return self
def next(self):
return "next"
def __next__(self):
return "__next__"
x = test()
for i,thing in enumerate(x):
print(thing)
if i>4:
break
in python 2 next is printed but in python 3 __next__ is printed, since the method is being called implicitly it makes way more sense to match other implicit methods such as __add__ or __getitem__, which is described in the PEP.
If you are planning on using next explicitly then much like len, iter, getattr, hash etc. then python provides a built in function to call the implicit function for you. At least... after PEP 3114. 😀
Also, the next built-in allows you pass a default value if you don't want to throw an error if the iterable is finished which is frequently useful:
it = iter([])
x = next(it, "pls don't crash")
which couldn't really be standardized for a .next() method call. As well objects such as a random number generator may define a .next (next number) method without necessarily wanting to be an iterator which could have left ambiguity / confusion.
Or maybe I should say, ways to skip having to initialize at all.
I really hate that every time I want to do a simple count variable, I have to say, "hey python, this variable starts at 0." I want to be able to say count+=1and have it instantly know to start from 0 at the first iteration of the loop. Maybe there's some sort of function I can design to accomodate this? count(1) that adds 1 to a self-created internal count variable that sticks around between iterations of the loop.
I have the same dislike for editing strings/lists into a new string/list.
(Initializing new_string=""/new_list=[] before the loop).
I think list comprehensions may work for some lists.
Does anyone have some pointers for how to solve this problem? I am fairly new, I've only been programming off and on for half a year.
Disclaimer: I do not think that this will make initialization any cleaner. Also, in case you have a typo in some uses of your counter variable, you will not get a NameError but instead it will just silently create and increment a second counter. Remember the Zen of Python:
Explicit is better than implicit.
Having said that, you could create a special class that will automatically add missing attributes and use this class to create and auto-initialize all sorts of counters:
class Counter:
def __init__(self, default_func=int):
self.default = default_func
def __getattr__(self, name):
if name not in self.__dict__:
self.__dict__[name] = self.default()
return self.__dict__[name]
Now you can create a single instance of that class to create an arbitrary number of counters of the same type. Example usage:
>>> c = Counter()
>>> c.foo
0
>>> c.bar += 1
>>> c.bar += 2
>>> c.bar
3
>>> l = Counter(list)
>>> l.blub += [1,2,3]
>>> l.blub
[1, 2, 3]
In fact, this is similar to what collections.defaultdict does, except that you can use dot-notation for accessing the counters, i.e. c.foo instead of c['foo']. Come to think of it, you could even extend defaultdict, making the whole thing much simpler:
class Counter(collections.defaultdict):
def __getattr__(self, name):
return self[name]
If you are using a counter in a for loop you can use enumerate:
for counter, list_index in enumerate(list):
the counter is the first variable in the statement and 1 is added to it per iteration of the loop, the next variable is the value of that iteration in the list. I hope this answers your first question as for your second, the following code might help
list_a = ["this", "is"]
list_b = ["a", "test"]
list_a += list_b
print(list_a)
["this", "is", "a", "test"]
The += works for strings as well because they are essentially lists aw well. Hope this helps!
It is apparently Pythonic to return values that can be treated as 'False' versions of the successful return type, such that if MyIterableObject: do_things() is a simple way to deal with the output whether or not it is actually there.
With generators, bool(MyGenerator) is always True even if it would have a len of 0 or something equally empty. So while I could write something like the following:
result = list(get_generator(*my_variables))
if result:
do_stuff(result)
It seems like it defeats the benefit of having a generator in the first place.
Perhaps I'm just missing a language feature or something, but what is the pythonic language construct for explicitly indicating that work is not to be done with empty generators?
To be clear, I'd like to be able to give the user some insight as to how much work the script actually did (if any) - contextual snippet as follows:
# Python 2.7
templates = files_from_folder(path_to_folder)
result = list(get_same_sections(templates)) # returns generator
if not result:
msg("No data to sync.")
sys.exit()
for data in result:
for i, tpl in zip(data, templates):
tpl['sections'][i]['uuid'] = data[-1]
msg("{} sections found to sync up.".format(len(result)))
It works, but I think that ultimately it's a waste to change the generator into a list just to see if there's any work to do, so I assume there's a better way, yes?
EDIT: I get the sense that generators just aren't supposed to be used in this way, but I will add an example to show my reasoning.
There's a semi-popular 'helper function' in Python that you see now and again when you need to traverse a structure like a nested dict or what-have-you. Usually called getnode or getn, whenever I see it, it reads something like this:
def get_node(seq, path):
for p in path:
if p in seq:
seq = seq[p]
else:
return ()
return seq
So in this way, you can make it easier to deal with the results of a complicated path to data in a nested structure without always checking for None or try/except when you're not actually dealing with 'something exceptional'.
mydata = get_node(my_container, ('path', 2, 'some', 'data'))
if mydata: # could also be "for x in mydata", etc
do_work(mydata)
else:
something_else()
It's looking less like this kind of syntax would (or could) exist with generators, without writing a class that handles generators in this way as has been suggested.
A generator does not have a length until you've exhausted its iterations.
the only way to get whether it's got anything or not, is to exhaust it
items = list(myGenerator)
if items:
# do something
Unless you wrote a class with attribute nonzero that internally looks at your iterations list
class MyGenerator(object):
def __init__(self, items):
self.items = items
def __iter__(self):
for i in self.items:
yield i
def __nonzero__(self):
return bool(self.items)
>>> bool(MyGenerator([]))
False
>>> bool(MyGenerator([1]))
True
>>>