Deep understanding of generators in Python

Generator concept

The generator does not save the results in a series, but saves the state of the generator and returns a value each time it iterates until it encounters a StopIteration exception.

Generator syntax

Generator expression: The same as the list parsing syntax, but replace the [] of list parsing with ()
The things that generator expressions can do can basically be handled by list parsing, but when it needs to be processed When the sequence is relatively large, list parsing consumes more memory.

>>> gen = (x**2 for x in range(5))
>>> gen
at 0x0000000002FB7B40>
>>> for g in gen:
... print(g, end='-')
...
0-1-4-9-16-
>>> for x in [0,1,2 ,3,4,5]:
... print(x, end='-')
...
0-1-2-3-4-5-

Generator function: In the function if If the yield keyword appears, then the function is no longer an ordinary function, but a generator function.
But the generator function can produce an unlimited sequence, so the list cannot be processed at all. The function of
yield is to turn a function into a generator. A function with yield is no longer an ordinary function, and the Python interpreter will treat it as a generator.

The following is a generator function that can produce odd numbers infinitely.

def odd():
n=1
while True:
n+=2
odd_num = odd()
count = 0
for o in odd_num:
if count >=5: break
print( o)
Count +=1

Of course, similar effects can be achieved by manually writing iterators, but the generator is more intuitive and easy to understand

class Iter:

def __init__(self):
self.start=-1
DEF __iter __ (Self):
Return Self __next __ (Self):
Self.start += 2
Return Self.start
i = it ()
For Count in Range (5):
Print (NExt (i) )


Digression: The generator contains __iter() and next__() methods, so you can directly use for to iterate, but self-written Iter that does not include StopIteration can only iterate through manual loops. isinstance(odd_num, Iterator)

True

>>> from collections import Iterator

>>> isinstance(odd_num, Iterator) )

True
>>> iter(odd_num) is odd_num
True
>>> help(odd_num)
Help on generator object:

odd = class generator(object)
| Methods defined here:
|
| __iter__(self, /)
| Implement iter(self).
|
| __next__(self, /)
| As a result, now you can loop according to the Iterator method with confidence!

When the for loop is executed, each loop will execute the code inside the fab function. When it reaches yield b, the fab function returns an iteration value. In the next iteration, the code continues to execute from the next statement of yield b, and The function's local variables look exactly the same as before the last interruption, so the function continues execution until yield is encountered again. It looks like a function is interrupted several times by yield during normal execution, and each interruption returns the current iteration value through yield.

yield and return

In a generator, if there is no return, StopIteration will be returned when the function is completed by default;

>>> def g1():

... yield 1

...

>>> g=g1()

>>> next(g) #When next(g) is called for the first time, it will hang after executing the yield statement, so the program does not Execution ends.

1

>>> next(g) #The program tries to start execution from the next statement of the yield statement and finds that it has reached the end, so it throws a StopIteration exception.

Traceback (most recent call last):

File "", line 1, in

StopIteration
>>>


If return is encountered, if return is during execution, Then throw StopIteration directly to terminate the iteration.

>>> def g2():
... yield 'a'
... return
... yield 'b'
...
>>> g=g2()
>>> next(g) #The program stays at the position after executing the yield 'a' statement.
'a'
>>> next(g) #The program finds that the next statement is return, so a StopIteration exception is thrown, so that the yield 'b' statement will never be executed.
Traceback (most recent call last):
File "", line 1, in
StopIteration

If a value is returned after return, then this value is a description of the StopIteration exception, not a program The return value.

The generator has no way to use return to return a value.

>>> def g3():
... yield 'hello'
... return 'world'
...
>>> g=g3()
>> > next(g)
'hello'
>>> next(g)
Traceback (most recent call last):
File "", line 1, in
StopIteration: world

Methods supported by generator

>>> help(odd_num)
Help on generator object:

odd = class generator(object)
| Methods defined here:
......
| close(...)
| close() -> raise GeneratorExit inside generator.
|
| send(...)
| send(arg) -> send 'arg' into generator,
| return next yielded value or raise StopIteration.
|
| throw(...)
| throw(typ[,val[,tb]]) -> raise exception in generator,
| return next yielded value or raise StopIteration.
. .....

close()

Manually close the generator function, and subsequent calls will directly return the StopIteration exception.

>>> def g4():
... yield 1
... yield 2
... yield 3
...
>>> g=g4()
> >> next(g)
1
>>> g.close()
>>> next(g) #After closing, the yield 2 and yield 3 statements will no longer work
Traceback ( most recent call last):
File "", line 1, in
StopIteration

send()

The biggest feature of the generator function is that it can accept a variable passed in from the outside, and The result is calculated based on the variable content and returned.
This is the most difficult part to understand about the generator function, and it is also the most important part. The implementation of the coroutine I will talk about later depends on it.

def gen():
value=0
while True:
receive=yield value
if receive=='e':
break
value = 'got: %s' % receive

g=gen()
print(g.send(None))
print(g.send('aaa'))
print(g.send(3))
print(g.send('e'))

Execution process:

The generator function can be started through g.send(None) or next(g) and executed to the end of the first yield statement.
At this time, the yield statement has been executed, but the receive has not been assigned a value.
yield value will output the initial value 0
Note: You can only send(None) when starting the generator function. If you try to enter other values, you will get an error message.

Through g.send('aaa'), aaa will be passed in and assigned to receive. Then the value of value will be calculated and returned to the while head. The execution of the yield value statement will stop.
At this time, the yield value will output "got: aaa" and then hang.

Through g.send(3), step 2 will be repeated, and the final output result is "got: 3"

When we g.send('e'), the program will execute break and then exit the loop, and finally The entire function is executed, so you will get a StopIteration exception.
The final execution result is as follows:

0
got: aaa
got: 3
Traceback (most recent call last):
File "h.py", line 14, in
print(g.send( 'e'))
StopIteration

throw()

is used to send an exception to the generator function, which can end system-defined exceptions or custom exceptions.
Throw() directly throws an exception and ends the program, or consumes a yield, or proceeds directly to the end of the program when there is no next yield.

)Def Gen ():

WHILE TRUE:
Try:
yield 'normal value'
yield' normal value 2'
print ('hee')
Except Valueerror:
Print ('We Got Valueerror Heres ') X Except Typeerror ; The result is:

normal value
we got ValueError here
normal value
normal value 2
Traceback (most recent call last):
File "h.py", line 15, in
print(g.throw (TypeError))
StopIteration

Explanation:

print(next(g)): will output normal value and stay before yield 'normal value 2'.


Since g.throw(ValueError) is executed, all subsequent try statements will be skipped, which means that yield 'normal value 2' will not be executed, and then enter the except statement and print out we got ValueError here.
Then enter the while statement part again and consume a yield, so the normal value will be output.


print(next(g)) will execute the yield 'normal value 2' statement and stay at the position after executing the statement.

g.throw(TypeError): will jump out of the try statement, so print('here') will not be executed, and then execute the break statement, jump out of the while loop, and then reach the end of the program, so a StopIteration exception will be thrown.

The following is a comprehensive example to expand a multi-dimensional list, or flatten a multi-dimensional list)

def flatten(nested):

try:
#If it is a string, manually throw TypeError. An If isInstance (Nested, Str):

Raise Typerror

for Sublist in Nested:
#YIELD FLATTEN (SUBLIST)

for Element in Flatten:

#Yield Element

Print ('GOT : ', Element)

Except TypeError:

          #print('here')

          yield nested
             
L=['aaadf',[1,2,3],2,4,[5,[6,[8,[9]],'ddf '],7]]
for num in flatten(L):
print(num)


If it is a bit difficult to understand, then it will be clearer to open the comment of the print statement and check it. The function generated by

yield from

yield is an iterator, so we usually put it in a loop statement to output the result.
Sometimes we need to put the iterator generated by this yield in another generator function, that is, generator nesting.
For example, the following example:

def inner():
for i in range(10):
yield i
def outer():

g_inner=inner() #This is a generator

while True:

res = g_Inner.Send (None)

yield res

g_outer = OUTER ()
While True:
Print (g_outer.Send (None))

Except Stoption:

Break


At this time, we can use yield from g. statement to reduce our workload.

def outer2():
Yield from inner()


Of course, the focus of the yield from statement is to help us automatically handle exceptions between the inner and outer layers. There are 2 well-written articles here, so I will No more nagging.
http://blog.theerrorlog.com/yield-from-in-python-3.html
http://stackoverflow.com/questions/9708902/in-practice-what-are-the-main-uses-for -the-new-yield-from-syntax-in-python-3

Summary

According to the duck model theory, the generator is a kind of iterator, which can be iterated using for.

When next(generator) is executed for the first time, the program will be suspended after executing the yield statement, and all parameters and status will be saved.

When next(generator) is executed again, it will be executed from the suspended state.
The loop ends when it encounters the end of the program or encounters StopIteration.

You can pass in parameters through generator.send(arg), which is the coroutine model.

You can pass in an exception through generator.throw(exception). The throw statement consumes a yield.

You can manually close the generator through generator.close().

next() is equivalent to send(None)

The above is the in-depth understanding of generators in Python. For more related articles, please pay attention to the PHP Chinese website (www.php.cn)!