Trap analysis of Python dynamic assignment

This article brings you an analysis of the pitfalls of dynamic assignment in Python. It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.

The namespace and scope issues may seem trivial, but in fact there is a lot behind them.

Due to space limitations, there is still an important knowledge content that has not been discussed, namely "the reading and writing issues of locals() and globals()". The reason why this issue is important is that it can implement some flexible dynamic assignment functions.

They are all dictionary types, and their usage is self-explanatory. However, there is a pitfall to be aware of when using it: globals() is readable and writable, while locals() is only readable but not writable. The article I shared today is to explore this issue. It is very in-depth and I would like to share it with you.

At work, sometimes we encounter a situation: dynamic variable assignment, whether it is a local variable or a global variable. When we are racking our brains, Python has solved this problem for us.

Python’s namespace is reflected in the form of a dictionary, and the specific functions are locals() and globals(), which correspond to the local namespace and global namespace respectively. Therefore, we can Through these methods, we can realize our "dynamic assignment" requirements.

For example:

def test():
    globals()['a2'] = 4
test()
print a2   # 输出 4

It is natural that since globals can change the global namespace, then of course locals should also be able to modify the local namespace. Modify the local variables within the function.

But is this really the case? No!

def aaaa():
    print locals()
    for i in ['a', 'b', 'c']:
        locals()[i] = 1
    print locals()
    print a
aaaa()

Output:

{}
{'i': 'c', 'a': 1, 'c': 1, 'b': 1}
Traceback (most recent call last):
  File "5.py", line 17, in <module>
    aaaa()
  File "5.py", line 16, in aaaa
    print a
NameError: global name &#39;a&#39; is not defined

The program runs and an error is reported!

But in The second time you print locals(), you can clearly see that there are already those variables in the local space, and there is also variable a with a value of 1. But why does a NameError exception appear when printing a?

More Look at an example:

def aaaa():
    print locals()
    s = 'test'                    # 加入显示赋值 s       
    for i in ['a', 'b', 'c']:
        locals()[i] = 1
    print locals()
    print s                       # 打印局部变量 s 
    print a
aaaa()

Output:

{}
{'i': 'c', 'a': 1, 's': 'test', 'b': 1, 'c': 1}
test
Traceback (most recent call last):
  File "5.py", line 19, in <module>
    aaaa()
  File "5.py", line 18, in aaaa
    print a
NameError: global name &#39;a&#39; is not defined

The upper and lower pieces of code. The difference is that the following code shows the assignment. Although the NameError exception is also triggered, the value of the local variable s is was printed.

This makes us feel very puzzled. Is changing local variables through locals() different from direct assignment? To solve this problem, we can only look at the truth of the program running, and Get the big killer dis~

Exploring the root cause

Directly analyze the second piece of code:

13           0 LOAD_GLOBAL              0 (locals)
              3 CALL_FUNCTION            0
              6 PRINT_ITEM
              7 PRINT_NEWLINE
 14           8 LOAD_CONST               1 ('test')
             11 STORE_FAST               0 (s)
 15          14 SETUP_LOOP              36 (to 53)
             17 LOAD_CONST               2 ('a')
             20 LOAD_CONST               3 ('b')
             23 LOAD_CONST               4 ('c')
             26 BUILD_LIST               3
             29 GET_ITER
        >>   30 FOR_ITER                19 (to 52)
             33 STORE_FAST               1 (i)
 16          36 LOAD_CONST               5 (1)
             39 LOAD_GLOBAL              0 (locals)
             42 CALL_FUNCTION            0
             45 LOAD_FAST                1 (i)
             48 STORE_SUBSCR
             49 JUMP_ABSOLUTE           30
        >>   52 POP_BLOCK
 17     >>   53 LOAD_GLOBAL              0 (locals)
             56 CALL_FUNCTION            0
             59 PRINT_ITEM
             60 PRINT_NEWLINE
 18          61 LOAD_FAST                0 (s)
             64 PRINT_ITEM
             65 PRINT_NEWLINE
 19          66 LOAD_GLOBAL              1 (a)
             69 PRINT_ITEM
             70 PRINT_NEWLINE
             71 LOAD_CONST               0 (None)
             74 RETURN_VALUE
None

You can see it in the above bytecode:

# The bytecode corresponding to ##locals() is: LOAD_GLOBAL

s='test'The corresponding bytecode is: LOAD_CONST and STORE_FAST

print sThe corresponding bytecode is: LOAD_FAST

The bytecode corresponding to print a is: LOAD_GLOBAL

As can be seen from the bytecodes of several key statements listed above, direct assignment/reading and assignment through locals() /The nature of reading is very different. So when the NameError exception is triggered, does it prove that the value stored through locals()[i] = 1 and the real local namespace are two different locations?

Think To answer this question, we must first determine one thing, that is, how to obtain the real local namespace? In fact, the standard answer to this question has been given in the above bytecode!

True local namespace The namespace actually exists in the corresponding data structure STORE_FAST. What the hell is this? This requires source code to answer:

// ceval.c  从上往下, 依次是相应函数或者变量的定义
// 指令源码
TARGET(STORE_FAST)
{
    v = POP();
    SETLOCAL(oparg, v);
    FAST_DISPATCH();
}
--------------------
// SETLOCAL 宏定义      
#define SETLOCAL(i, value)      do { PyObject *tmp = GETLOCAL(i); \
                                     GETLOCAL(i) = value; \
                                     Py_XDECREF(tmp); } while (0)
-------------------- 
// GETLOCAL 宏定义                                    
#define GETLOCAL(i)     (fastlocals[i])     
-------------------- 
// fastlocals 真面目
PyObject * PyEval_EvalFrameEx(PyFrameObject *f, int throwflag){
    // 省略其他无关代码
   fastlocals = f->f_localsplus;
....
}

After seeing this, it should be clear that the local namespace inside the function actually Yes, it is the member f_localsplus of the frame object. This is an array. It may be clearer to understand the children's shoes created by the function. During CALL_FUNCTION, this array will be initialized, and the formal parameter assignments will be stuffed in order. In the word Section code 18 61 LOAD_FAST 0 (s), the 0 in the fourth column is to take out the 0th member of f_localsplus, which is the value "s".

So STORE_FAST is the real way to store the variable Local namespace, what the hell is locals()? Why does it look like the real thing?

This requires analyzing locals. For this, bytecode may not be helpful. How about directly looking at the built-in functions? Define it:

// bltinmodule.c
static PyMethodDef builtin_methods[] = {
    ...
    // 找到 locals 函数对应的内置函数是 builtin_locals 
    {"locals",          (PyCFunction)builtin_locals,     METH_NOARGS, locals_doc},
    ...
}
-----------------------------
// builtin_locals 的定义
static PyObject *
builtin_locals(PyObject *self)
{
    PyObject *d;
    d = PyEval_GetLocals();
    Py_XINCREF(d);
    return d;
}
-----------------------------
PyObject *
PyEval_GetLocals(void)
{
    PyFrameObject *current_frame = PyEval_GetFrame();  // 获取当前堆栈对象
    if (current_frame == NULL)
        return NULL;
    PyFrame_FastToLocals(current_frame); // 初始化和填充 f_locals
    return current_frame->f_locals;
}
-----------------------------
// 初始化和填充 f_locals 的具体实现
void
PyFrame_FastToLocals(PyFrameObject *f)
{
    /* Merge fast locals into f->f_locals */
    PyObject *locals, *map;
    PyObject **fast;
    PyObject *error_type, *error_value, *error_traceback;
    PyCodeObject *co;
    Py_ssize_t j;
    int ncells, nfreevars;
    if (f == NULL)
        return;
    locals = f->f_locals;
    // 如果locals为空, 就新建一个字典对象
    if (locals == NULL) {
        locals = f->f_locals = PyDict_New();  
        if (locals == NULL) {
            PyErr_Clear(); /* Can't report it :-( */
            return;
        }
    }
    co = f->f_code;
    map = co->co_varnames;
    if (!PyTuple_Check(map))
        return;
    PyErr_Fetch(&error_type, &error_value, &error_traceback);
    fast = f->f_localsplus;
    j = PyTuple_GET_SIZE(map);
    if (j > co->co_nlocals)
        j = co->co_nlocals;
    // 将 f_localsplus 写入 locals
    if (co->co_nlocals)
        map_to_dict(map, j, locals, fast, 0);
    ncells = PyTuple_GET_SIZE(co->co_cellvars);
    nfreevars = PyTuple_GET_SIZE(co->co_freevars);
    if (ncells || nfreevars) {
        // 将 co_cellvars 写入 locals
        map_to_dict(co->co_cellvars, ncells,
                    locals, fast + co->co_nlocals, 1);
        if (co->co_flags & CO_OPTIMIZED) {
            // 将 co_freevars 写入 locals
            map_to_dict(co->co_freevars, nfreevars,
                        locals, fast + co->co_nlocals + ncells, 1);
        }
    }
    PyErr_Restore(error_type, error_value, error_traceback);
}

As you can see from PyFrame_FastToLocals above, locals() actually does the following things:

Determine whether the f_f->f_locals of the frame object is empty. If so, Then create a new dictionary object.

Write localsplus, co_cellvars and co_freevars into f_f->f_locals.

Here is a brief introduction to what the above are:

localsplus: Function parameters (positional parameter keyword parameters), display assigned variables.

co_cellvars and co_freevars: local variables used by closure functions.

Conclusion

Through the above source code, we already know clearly that what locals() sees is indeed the content of the local namespace of the function, but it itself cannot represent the local namespace. It is like a proxy, which collects A, B , C's stuff is shown to me, but I can't simply change this proxy to change what A, B, and C really own!

That's why, when we pass locals() When dynamically assigning values ​​using [i] = 1, print a triggers a NameError exception. On the contrary, globals() is indeed the real global namespace, so it is generally said: locals() is read-only and globals() is readable. Writable

This article has ended here. For more exciting content, you can pay attention to the python video tutorial column on the PHP Chinese website!

The above is the detailed content of Trap analysis of Python dynamic assignment. For more information, please follow other related articles on the PHP Chinese website!