Miscellaneous talk on the use of Iterator in Python

Iterator is an object that supports next() operation. It contains a set of elements. When the next() operation is performed, one of the elements is returned; when all elements are returned, a StopIteration exception is generated.

>>>a=[1,2,3]
>>>ia=iter(a)
>>>next(ia)
1
>>>next(ia)
2
>>>next(ia)
3
>>>next(ia)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
StopIteration

ite() can accept a variety of Python objects as parameters, such as list, tuple, dict, set, etc., and convert them into iterators. Iterators can be used in for statements or in statements. Many common operations also support iterators, such as sum(), max(), etc.

>>> b=[4,5,6]
>>> ib=iter(b)
>>> for x in ib:
...   print(x)
...
4
5
6
>>> ic=iter(b)
>>> sum(ic)
15
>>> id=iter(b)
>>> max(ic)
6

Needless to say, iterators have many benefits:

1. "Streaming" data processing method reduces memory consumption:
For example, when processing a file, suddenly taking out all the data and putting it into the memory for processing will cause the program to consume a lot of memory, and sometimes it is even impossible to do. Generally, we will process the file content part by part:

for text_line in open("xx.txt"):
 print text_line

2. Or when processing xml files:

tree = etree.iterparse(xml, ['start', 'end'])
for event, elem in tree:
  if event == "end"
    result = etree.tostring(elem)
    elem.clear()
    print result

The file object returned by the built-in function open and the xml tree serialized by etree.iterparse are both iterable objects, which allow us to progressively process the contents of the file.

3. Supports the convenience of using for statements to consume data:
Some common built-in types in Python such as arrays, lists and even strings are iterable types, so that we can use the syntax sugar of the for statement to conveniently consume data without having to record the index position ourselves. Human loops :

for i in [1, 2, 3, 4]
 print i,

After briefly understanding the benefits of iterators, let’s talk about python’s iterator mode in a serious way.
Here we introduce two more convoluted terms: iterable objects and iterator objects. Personally, I feel that starting from these two concepts will give a better understanding of iterators. Before giving examples, let’s give a rough explanation of these two concepts:

Iterable object: The object contains the implementation of the __iter()__ method. After the iter function of the object is called, it will return an iterator, which contains the implementation of specific data acquisition.
Iterator: Contains the implementation of the next method, returns the expected data within the correct range, and can throw a StopIteration error to stop iteration after exceeding the range.
Give me an example and say it while reading:

class iterable_range:
  def __init__(self, n):
    self.n = n

  def __iter__(self):
    return my_range_iterator(self.n)

class my_range_iterator:
  def __init__(self, n):
    self.i = 0
    self.n = n

  def next(self):
    if self.i < self.n:
      i = self.i
      self.i += 1
      print 'iterator get number:', i
      return i
    else:
      raise StopIteration()

The iterable_range in the example is an iterable object, so we can also use the for statement to iterate over it:

temp = my_range(10)
for item in temp:
  print item,

Output:

  my iterator get number: 0
  0
  my iterator get number: 1
  1
  my iterator get number: 2
  2
  my iterator get number: 3
  3
  my iterator get number: 4
  4
  my iterator get number: 5
  5
  my iterator get number: 6
  6
  my iterator get number: 7
  7
  my iterator get number: 8
  8
  my iterator get number: 9
  9

You can take a closer look at the output log:

• The data is indeed “streamed”
• Iterator is the person who really does the work behind the scenes
• The for statement can iterate the data of an object very conveniently.

Iterable objects are actually more like the upper layer of the entire iterator pattern, like a constraint, a contract, and a specification. It can ensure that it can return an iterator object that does actual work. Methods such as for and sum that accept an iterable object all follow this specification: call the __iter__ function of the object, return the iterator, process each value returned by the iterator object, or require some summary operations. Take for as an example:

iterator_object = iterable_object.__iter__()
while True:
  try:
    value = iterator_object.next()
  except StopIteration:
    # StopIteration exception is raised after last element
    break

  # loop code
  print value

The logic behind the syntax sugar of

for is almost as shown in the code in the above example: first get the iterator object returned by the iterable object, and then call the next method of the iterator object to get each value. In the process of getting the value Detect the boundary at any time - that is, check whether an error such as StopIteration is thrown. If the iterator object throws an error, the iteration stops (note: As can be seen from this example, for those methods that accept iterable objects, if we pass an In fact, a simple iterator object cannot work, and an error similar to TypeError: iteration over non-sequence may be reported).
Of course, generally we will not separate them intentionally during the application process. We can slightly modify the iterator object and add the implementation of the __iter__ method, so that the object itself is both an iterable object and an iterator object. :

class my_range_iterator:
   def __init__(self, n)：
    self.i = 0
    self.n = n

   def __iter__(self):
    return self

   def next(self):
    if self.i < self.n:
      i = self.i

      self.i += 1
      print 'my iterator get number:', i
      return i
    else:
      raise StopIteration()

 for item in my_range_iterator(10):
   print item

Output:

  my iterator get number: 0
  0
  my iterator get number: 1
  1
  my iterator get number: 2
  2
  my iterator get number: 3
  3
  my iterator get number: 4
  4
  my iterator get number: 5
  5
  my iterator get number: 6
  6
  my iterator get number: 7
  7
  my iterator get number: 8
  8
  my iterator get number: 9
  9