Python Advanced: Detailed Description of Higher-Order Functions

This article talks about Python advanced: detailed description of high-order functions. Friends in need can refer to

Functional programming

Function is a kind of encapsulation supported by Python's built-in support. By splitting large sections of code into functions and calling functions layer by layer, we can decompose complex tasks into simple tasks. This decomposition can be called process-oriented programming. Functions are the basic unit of process-oriented programming.

Functional Programming (please note the additional word "formula") - Functional Programming, although it can also be attributed to process-oriented programming, its ideas are closer to mathematical calculations.

We must first understand the concepts of computer and computing.

At the computer level, the CPU executes instruction codes for addition, subtraction, multiplication and division, as well as various conditional judgments and jump instructions. Therefore, assembly language is the language closest to the computer.

Computing refers to calculations in the mathematical sense. The more abstract the calculation, the farther away it is from the computer hardware.

corresponds to programming languages, that is, the lower-level language is closer to the computer, has a lower level of abstraction, and has higher execution efficiency, such as C language; the higher-level language is closer to computing, has a higher level of abstraction, and has lower execution efficiency. , such as Lisp language.

Functional programming is a programming paradigm with a high degree of abstraction. Functions written in pure functional programming languages ​​have no variables. Therefore, for any function, as long as the input is certain, the output will be certain. This A pure function is said to have no side effects. In programming languages ​​that allow the use of variables, since the variable status inside the function is uncertain, the same input may result in different outputs. Therefore, this kind of function has side effects.

One feature of functional programming is that it allows the function itself to be passed into another function as a parameter, and it also allows a function to be returned!

Python provides partial support for functional programming. Since Python allows the use of variables, Python is not a purely functional programming language.

Higher-order function

Higher-order function is called Higher-order function in English. What are higher-order functions? We use actual code as an example to deepen the concepts step by step.

Variables can point to functions

Take Python’s built-in absolute value function abs() as an example. Use the following code to call this function:

>>> abs(-10)10

But what if you only write abs?

>>> abs

It can be seen that abs(-10) is a function call, and abs is the function itself.

To get the function call result, we can assign the result to a variable:

>>> x = abs(-10)>>> x10

But what if the function itself is assigned to a variable?

>>> f = abs
>>> f

Conclusion: The function itself can also be used Assign a value to a variable, that is: the variable can point to a function.

If a variable points to a function, can the function be called through the variable? Verify it with the code:

>>> f = abs>>> f(-10)10

Success! It means that the variable f now points to the abs function itself. Calling the abs() function directly is exactly the same as calling the variable f().

The function name is also a variable

So what is the function name? The function name is actually the variable pointing to the function! For the function abs(), the function name abs can be regarded as a variable, which points to a function that can calculate the absolute value!

What will happen if abs points to other objects?

>>> abs = 10
>>> abs(-10)
Traceback (most recent call last):
File "" , line 1, in
TypeError: 'int' object is not callable

After pointing abs to 10, the function cannot be called through abs(-10)! Because the abs variable no longer points to the absolute value function but to an integer 10!

Of course, the actual code must not be written like this. This is to illustrate that the function name is also a variable. To restore the abs function, restart the Python interactive environment.

Note: Since the abs function is actually defined in the import builtins module, to modify the pointer of the abs variable to take effect in other modules, use import builtins; builtins.abs = 10.

Passing in function

Since variables can point to functions and function parameters can receive variables, then a function can receive another function as a parameter. This kind of function is called a higher-order function. .

The simplest higher-order function:

def add(x, y, f): return f(x) + f(y)

When When we call add(-5, 6, abs), the parameters x, y and f receive -5, 6 and abs respectively. According to the function definition, we can deduce the calculation process as:

x = -5
y = 6
f = abs
f(x) + f(y) ==> abs(-5) + abs(6) ==> 11return 11

Verify it with code:

>>> add(-5, 6, abs)11

Writing a high-order function is to allow the parameters of the function to be received other functions.

Summary

Passing in a function as a parameter is called a higher-order function. Functional programming refers to this highly abstract programming paradigm.

map/reduce

Python has built-in map() and reduce() functions.

If you have read Google's famous paper "MapReduce: Simplified Data Processing on Large Clusters", you can roughly understand the concept of map/reduce.

Let’s look at the map first. The map() function receives two parameters, one is a function and the other is an Iterable. map applies the passed function to each element of the sequence in turn and returns the result as a new Iterator.

For example, for example, we have a function f(x)=x2, and we want to apply this function to a list [1, 2, 3, 4, 5, 6, 7, 8, 9]. You can use map() to implement it as follows:

Now, we use Python code to implement it:

>>> def f(x): ... return x * x
...>>> r = map(f, [1, 2, 3, 4, 5, 6, 7, 8, 9])>>> list(r)
[1, 4, 9, 16, 25, 36, 49, 64, 81]

The first parameter passed in to map() is f, which is the function the object itself. Since the result r is an Iterator, and Iterator is a lazy sequence, it uses the list() function to let it calculate the entire sequence and return a list.

You may think that you don’t need the map() function. You can also calculate the result by writing a loop:

L = []for n in [1, 2, 3, 4, 5, 6, 7, 8, 9]:
L.append(f(n))
print(L)

is indeed possible, but from the above loop code, Can you understand at a glance "apply f(x) to each element of the list and generate a new list as a result"?

So, map(), as a high-order function, actually abstracts the operation rules. Therefore, we can not only calculate simple f(x)=x2, but also calculate any complex function, such as, Convert all the numbers in this list to strings:

>>> list(map(str, [1, 2, 3, 4, 5, 6, 7, 8, 9]))[ '1', '2', '3', '4', '5', '6', '7', '8', '9']

Only one line of code is required.

Let’s look at the usage of reduce. Reduce applies a function to a sequence [x1, x2, x3, ...]. This function must receive two parameters. Reduce continues the cumulative calculation of the result with the next element of the sequence. The effect is:

reduce(f, [x1, x2, x3, x4]) = f(f(f(x1, x2), x3), x4)

For example, sum a sequence , you can use reduce to implement:

>>> from functools import reduce>>> def add(x, y):... return x + y
...> ;>> reduce(add, [1, 3, 5, 7, 9])25

Of course, the sum operation can be directly performed using Python’s built-in function sum(), there is no need to use reduce .

But if you want to transform the sequence [1, 3, 5, 7, 9] into the integer 13579, reduce can come in handy:

>>> from functools import reduce> ;>> def fn(x, y):... return x * 10 + y
...>>> reduce(fn, [1, 3, 5, 7, 9]) 13579

This example itself is not very useful, but if we consider that the string str is also a sequence, we can slightly change the above example and cooperate with map(), we can write the conversion of str Function for int:

>>> from functools import reduce>>> def fn(x, y):... return x * 10 + y
...> ;>> def char2num(s):... return {'0': 0, '1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9}[s]
...>>> reduce(fn, map(char2num, ' 13579'))13579

The function organized into a str2int is:

from functools import reducedef str2int(s): def fn(x, y): return x * 10 + y def char2num(s): return {'0': 0, '1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9}[s] return reduce( fn, map(char2num, s))

can also be further simplified using the lambda function:

from functools import reducedef char2num(s): return {'0': 0, '1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9 ': 9}[s]def str2int(s): return reduce(lambda x, y: x * 10 + y, map(char2num, s))

That is, assuming Python does not Providing the int() function, you can write a function to convert a string into an integer yourself, and it only requires a few lines of code!

The usage of lambda function will be introduced later.

filter

Python's built-in filter() function is used to filter sequences.

Similar to map(), filter() also receives a function and a sequence. Unlike map(), filter() applies the passed function to each element in turn, and then decides to retain or discard the element based on whether the return value is True or False.

For example, in a list, to delete even numbers and keep only odd numbers, you can write:

def is_odd(n): return n % 2 == 1

list(filter(is_odd, [1, 2, 4, 5, 6, 9, 10, 15]))# Result: [1, 5, 9, 15]

Put a sequence into To delete the empty string, you can write:

def not_empty(s): return s and s.strip()

list(filter(not_empty, ['A', '' , 'B', None, 'C', ' ']))# Result: ['A', 'B', 'C']

It can be seen that the higher-order function filter() is used , the key is to correctly implement a "filter" function.

Note that the filter() function returns an Iterator, which is a lazy sequence, so to force filter() to complete the calculation results, you need to use the list() function to obtain all the results and return the list.

sorted

Sort algorithm

Sort is also an algorithm often used in programs. Whether bubble sort or quick sort is used, the core of sorting is to compare the sizes of two elements. If it's a number, we can compare it directly, but what if it's a string or two dicts? Direct comparison of mathematical magnitudes is meaningless, therefore, the comparison process must be abstracted through functions.

Python’s built-in sorted() function can sort the list:

>>> sorted([36, 5, -12, 9, -21])[- 21, -12, 5, 9, 36]

In addition, the sorted() function is also a higher-order function. It can also receive a key function to implement customized sorting, such as by absolute value Size sorting:

>>> sorted([36, 5, -12, 9, -21], key=abs)
[5, 9, -12, -21, 36 ]

The function specified by key will act on each element of the list and be sorted according to the result returned by the key function. Compare the original list and the list processed by key=abs:

list = [36, 5, -12, 9, -21]keys = [36, 5, 12, 9, 21]

Then the sorted() function sorts according to keys and returns the corresponding elements of the list according to the corresponding relationship:

keys sorting result=> [5, 9, 12, 21, 36]
| |
#>>> sorted(['bob', 'about', 'Zoo', 'Credit'])['Credit', 'Zoo', 'about', 'bob']

By default, string sorting is based on ASCII size comparison. Since 'Z'

Now, we propose that sorting should ignore case and sort in alphabetical order. To implement this algorithm, there is no need to make major changes to the existing code, as long as we can use a key function to map the string to a case-ignoring sort. Comparing two strings by ignoring case actually means changing the strings to uppercase (or lowercase) first, and then comparing them.

In this way, we pass the key function to sorted to achieve sorting that ignores case:

>>> sorted(['bob', 'about', 'Zoo ', 'Credit'], key=str.lower)

['about', 'bob', 'Credit', 'Zoo']

To perform reverse sorting, no changes are required key function, you can pass in the third parameter reverse=True:

>>> sorted(['bob', 'about', 'Zoo', 'Credit'], key=str. lower, reverse=True)
['Zoo', 'Credit', 'bob', 'about']

As can be seen from the above examples, the abstraction ability of high-order functions is very Powerful, and the core code can be kept very simple.

summary

sorted() is also a higher-order function. The key to sorting with sorted() is to implement a mapping function.

Return function

Function as return value

In addition to accepting functions as parameters, higher-order functions can also return functions as result values.

Let’s implement the summation of a variable parameter. Usually, the sum function is defined like this:

def calc_sum(*args):
ax = 0 for n in args:
ax = ax + n return ax

However, what if the sum does not need to be calculated immediately, but is calculated as needed in the subsequent code? Instead of returning the summation result, you can return the summation function:

def lazy_sum(*args): def sum():
ax = 0 for n in args:
ax = ax + n return ax return sum

When we call lazy_sum(), what is returned is not the summation result, but the summation function:

>>> f = lazy_sum(1, 3, 5, 7, 9)
>>> f
.sum at 0x101c6ed90>

Call The result of the summation is actually calculated when the function f is used:

>>> f()25

In this example, we define it in the function lazy_sum Function sum, and the internal function sum can refer to the parameters and local variables of the external function lazy_sum. When lazy_sum returns the function sum, the relevant parameters and variables are saved in the returned function. This is called "Closure" Program structure has great power.

Please note one more thing, when we call lazy_sum(), each call will return a new function, even if the same parameters are passed in:

>>> f1 = lazy_sum(1, 3, 5, 7, 9)>>> f2 = lazy_sum(1, 3, 5, 7, 9)>>> f1==f2False

## The calling results of #f1() and f2() do not affect each other.

Closure

Notice that the returned function refers to the local variable args within its definition, so when a function returns a function, its internal local variables are also referenced by the new function , so closure is easy to use, but not easy to implement.

Another issue that needs attention is that the returned function is not executed immediately, but is not executed until f() is called. Let’s look at an example:

def count():

fs = [] for i in range(1, 4): def f(): return i*i
fs.append( f) return fs

f1, f2, f3 = count()

In the above example, each time it loops, a new function is created, and then the created All three functions have returned.

You may think that the results of calling f1(), f2() and f3() should be 1, 4, 9, but the actual result is:

>>> f1() 9>>> f2()9>>> f3()9

All are 9! The reason is that the returned function refers to the variable i, but it is not executed immediately. By the time all three functions return, the variable i they refer to has become 3, so the final result is 9.

One thing to keep in mind when returning a closure is: the return function should not reference any loop variables, or variables that will change subsequently.

What if you must reference the loop variable? The method is to create another function and use the parameters of the function to bind the current value of the loop variable. No matter how the loop variable changes subsequently, the value bound to the function parameter remains unchanged:

def count(): def f (j): def g (): Return J*J Return g

FS = [] for I in Range (1, 4): ## FS.append (F (I))#f (i) is executed immediately, so the current value of i is passed into f() return fs


Look at the result again:

>>> f1, f2, f3 = count ()>>> f1()1>>> f2()4>>> f3()9

The disadvantage is that the code is long and can be shortened by using the lambda function code.

Summary

A function can return a calculation result or a function.

When returning a function, remember that the function has not been executed, and do not reference any variables that may change in the returned function.

Anonymous function

When we pass in a function, sometimes there is no need to explicitly define the function. It is more convenient to pass in the anonymous function directly.

In Python, there is limited support for anonymous functions. Still taking the map() function as an example, when calculating f(x)=x2, in addition to defining a function of f(x), you can also directly pass in an anonymous function:

>>> list (map(lambda x: x * x, [1, 2, 3, 4, 5, 6, 7, 8, 9]))[1, 4, 9, 16, 25, 36, 49, 64, 81]

It can be seen from comparison that the anonymous function lambda x: x * x is actually:

def f(x): return x * x

The keyword lambda represents an anonymous function, and the x before the colon represents function parameters.

Anonymous functions have a limitation, that is, they can only have one expression. There is no need to write return. The return value is the result of the expression.

There is an advantage to using anonymous functions, because the function has no name, so you don’t have to worry about function name conflicts. In addition, the anonymous function is also a function object. You can also assign the anonymous function to a variable and then use the variable to call the function:

>>> f = lambda x: x * x
> ;>> f
at 0x101c6ef28>
>>> f(5)
25

##Similarly, you can also put anonymous The function returns as a return value, for example:

def build(x, y): return lambda: x * x + y * y

Summary

Python pair Support for anonymous functions is limited, and only some simple situations can use anonymous functions.

Partial function

Python's functools module provides many useful functions, one of which is partial function. It should be noted that the partial function here is different from the partial function in the mathematical sense.

When introducing function parameters, we mentioned that by setting the default values ​​of parameters, the difficulty of function calling can be reduced. Partial functions can also do this. An example is as follows: The

int() function can convert a string into an integer. When only a string is passed in, the int() function defaults to decimal conversion:

>>> int('12345')12345

But the int() function also provides an additional base parameter, the default value is 10. If you pass in the base parameter, you can perform N-base conversion:

>>> int('12345', base=8)5349

>>> int(' 12345', 16)74565

Assuming that we want to convert a large number of binary strings, it is very troublesome to pass in int(x, base=2) every time, so we thought that we can define an int2( ) function, base=2 is passed in by default:

def int2(x, base=2): return int(x, base)

In this way, we convert binary Very convenient:

>>> int2('1000000')64>>> int2('1010101')85

functools.partial is to help us To create a partial function, we do not need to define int2() ourselves. We can directly use the following code to create a new function int2:

>>> import functools>>> int2 = functools .partial(int, base=2)>>> int2('1000000')64>>> int2('1010101')85

So, a brief summary of functools.partial The function is to fix certain parameters of a function (that is, set default values) and return a new function. It will be easier to call this new function.

Note that the new int2 function above only resets the base parameter to the default value of 2, but other values ​​can also be passed in when the function is called:

>> > int2('1000000', base=10)1000000

Finally, when creating a partial function, you can actually receive three parameters: function object, *args and **kw. When passed in :

int2 = functools.partial(int, base=2)

actually fixes the keyword parameter base of the int() function, that is:

int2('10010')

Equivalent to:

kw = { 'base': 2 }int('10010', **kw)

When passed in:

max2 = functools.partial(max, 10)

In fact, 10 will be automatically added to the left as part of *args, that is:

max2(5, 6, 7)

is equivalent to:

args = (10, 5, 6, 7)

max(*args )

The result is 10.

Summary

When a function has too many parameters and needs to be simplified, you can use functools.partial to create a new function. This new function can fix some of the parameters of the original function, thus It's simpler when calling.

The above is the detailed content of Python Advanced: Detailed Description of Higher-Order Functions. For more information, please follow other related articles on the PHP Chinese website!