how to make more expressive python iterators? just like c++ iterator - c++

Question

Firstly, I review the c++ style iterators quickly.for example:
//--- Iterating over vector with iterator.
vector<int> v;
. . .
for (vector<int>::iterator it = v.begin(); it!=v.end(); ++it) {
cout << *it << endl;
}
It is flexible. It is easily to change underlying container types. For example, you might decide later that the number of insertions and deletions is so high that a list would be more efficient than a vector. It also has many useful member functions. Many of the member functions for vector use iterators, for example, assign, insert, or erase. Moreover, we can use iterator (if supported) bidirectionaly, such as ++, --. This is useful to parse a stream like objects.
The problems of python is:
1:Currently, python for loop syntax is less flexible than c++ for. (well , safer)
2:rather than "it != iter.end()" style, python will throw exception when next() has no more. It is not flexible.
Question 1: Is my idea above correct?
OK. Here comes my question, how to implement a more powerful python iterator as powerful as c++ iterators? Currently, python for loop syntax is less flexible than c++ for. I also find some possible solutions, such as http://www.velocityreviews.com/forums/t684406-pushback-iterator.html. but it asks user to push_back a stuff rather than ask iterator --.
Question 2: What is the best to implement a Bidirectional Iterator in python? Just like http://www.cplusplus.com/reference/std/iterator/BidirectionalIterator/.
The pseudo-code is the following:
it = v.begin();
while( it!=v.end()) {
//do sth here
if (condition1)
++it;//suppose this iterator supports ++
if(condition2)
--it;//suppose this iterator supports --
}
The key features are: 1) bidirectional , 2) simpler "end" checking. The "++" or "--" operators or common functions do not matter (it has no semantic difference anyway).
Thanks,
Update:
I got some possible solutions from the answers:
i = 0
while i < len(sequence): # or i < len and some_other_condition
star_it = sequence[i]
if condition_one(star_it):
i += 1
if condition_two(star_it):
i = max(i - 1, 0)
However, unlike array, random access of list should be O(n). I suppose the "list" object in python internally is implemented using linked-list like stuff. Thus, this while loop solution is not efficient. However, in c++, we have "random iterator", "bidirectional iterator". How should I get a better solution? Thanks.

Answer 1

For the majority of situations, Python's for and iterators are the simplest thing around. That is their goal and they shouldn't compromise it for flexibility -- their lack of flexibility isn't a problem.
For a few situations where you couldn't use a for loop, C++ iterators might be simpler. But there is always a way to do it in Python that isn't much more complex than using a C++ iterator.
If you need to separate advancing the iterator from looping, just use a while loop:
it = iter(obj)
try:
while True: # or some secondary break condition other than StopIteration
star_it = next(it)
if condition_one(star_it):
star_it = next(it)
except StopIteration:
pass # exhausted the iterator
I can think of only two situations where --it makes sense in Python.
The first is you're iterating over a sequence. In that case, if you need to go backwards, don't use an iterator at all -- just use a counter with a while loop:
i = 0
while i < len(sequence): # or i < len and some_other_condition
star_it = sequence[i]
if condition_one(star_it):
i += 1
if condition_two(star_it):
i = max(i - 1, 0)
The second is if you're iterating over a doubly linked list. In that case, again, don't use an iterator -- just traverse the nodes normally:
current = node
while current: # or any break condition
if condition_one(current):
current = current.next
if condition_two(star_it):
current = current.prev
A situation where you might think it makes sense, but you can't use either of the above methods, is with an unordered collection like a set or dict. However, --it doesn't make sense in that case. As the collection is unordered, semantically, any of the items previously reached would be appropriate -- not just the actual previous item.
So, in order to know the right object to go back to, you need memory, either by iterating over a sequence like mydict.values() or tuple(myset) and using a counter, or by assembling a sequence of previous values as you go and using a while loop and next as above instead of a for loop.

Answer 2

Solutions for a few situations you mentioned:
You want to replace objects in the underlying container. For dictionaries, iterate over the keys or items, not only the values:
for key, value in my_dict.iteritems():
if conditiion(value):
my_dict[key] = new_value
For lists use enumerate():
for index, item in enumerate(my_list):
if condition(item):
my_list[index] = new_item
You want an iterator with one "look-ahead" value. You probably would use something tailored to a specific situation, but here's a recipe for general situations:
def iter_with look_ahead(iterable, sentinel=None):
iterable, it_ahead = itertools.tee(iterable)
next(it_ahead, None)
return izip_longest(iterable, it_ahead, fillvalue=sentinel)
for current, look_ahead in iter_with look_ahead(tokens):
# whatever
You want to iterate in reverse. Use reversed() for containers that support it.
You want random access. Just turn your iterable into a list and use indices:
my_list = list(my_iterable)

Answer 3

Actually, C++ iterator system is not so great. Iterators are akin to pointers, and they have their woes:
singular values: v.end() cannot be dereferenced safely
inversion issues: std::for_each(end, begin, func);
mismatch issues: std::for_each(v0.begin(), v2.end(), func);
Python approach is much better in this regard (though the use of exception can be quite surprising at first, it really helps defining nested iterators), because contrary to its name, a Python iterator is more akin to a Range.
The concept of Range is so much better than C++11 introduces the range-for loop construct:
for (Object& o: range) {
}
Anything that is possible with an iterator is also possible with a range, though it may take some times to realize it and some translations seem surrealists at first for those of us who were educated with C++ pointer-like iterators. For example, subranges can perfectly be expressed:
for (Object& o: slice(range, 2, 9)) {
}
where slice would take all elements in position [2, 9) within range.
So, instead of fighting your language (Python) you should delve further into it and embrace its style. Fighting against a language is generally a losing battle, learn its idioms, become efficient.

Answer 4

You could implement a similar way of C++ using python objects:
class Iterable(object):
class Iterator(object):
def __init__(self, father, pos=0):
self.father = father
self.pos = pos
def __getitem__(self, pos=0):
return self.father[self.pos + pos]
def __setitem__(self, pos, value):
self.father[self.pos + pos] = value
def __iadd__(self, increment):
self.pos += increment
return self
def __isub__(self, decrement):
self.pos -= decrement
return self
def __ne__(self, other):
return self.father != other.father or self.pos != other.pos
def __eq__(self, other):
return not (self != other)
def begin(self):
return self.Iterator(self)
def end(self):
return self.Iterator(self, len(self))
class Vector(list, Iterable):
pass
v = Vector([54, 43, 32, 21])
counter = 0
it = v.begin()
print it, it[0]
while it != v.end():
counter += 1
print it[0]
if counter == 2:
it += 1; # suppose this iterator supports ++
if counter == 1:
it -= 1; # suppose this iterator supports --
it += 1
This replaces *it by it[0] (also analog to C++) and it++ by it += 1, but in effect it stays pretty much the same.
You leave the Pythonic ways if you do this, though ;-)

Answer 5

Note that the list object in Python is an array, so the efficiency concern mentioned in the question is actually a non-issue.