search

Home  >  Q&A  >  body text

python - pandas内存错误

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

<code>  File "D:\WinPython-32bit-2.7.9.3\python-2.7.9\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 680, in runfile

    execfile(filename, namespace)

 

  File "D:\WinPython-32bit-2.7.9.3\python-2.7.9\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 71, in execfile

    exec(compile(scripttext, filename, 'exec'), glob, loc)

 

  File "D:/pywork/Marking/DataProcessing.py", line 113, in <module>

    df = pd.DataFrame(data, columns=colNames)

 

  File "D:\WinPython-32bit-2.7.9.3\python-2.7.9\lib\site-packages\pandas\core\frame.py", line 258, in __init__

    dtype=dtype)

 

  File "D:\WinPython-32bit-2.7.9.3\python-2.7.9\lib\site-packages\pandas\core\frame.py", line 4627, in _arrays_to_mgr

    return create_block_manager_from_arrays(arrays, arr_names, axes)

 

  File "D:\WinPython-32bit-2.7.9.3\python-2.7.9\lib\site-packages\pandas\core\internals.py", line 3539, in create_block_manager_from_arrays

    blocks = form_blocks(arrays, names, axes)

 

  File "D:\WinPython-32bit-2.7.9.3\python-2.7.9\lib\site-packages\pandas\core\internals.py", line 3614, in form_blocks

    int_blocks = _multi_blockify(int_items)

 

  File "D:\WinPython-32bit-2.7.9.3\python-2.7.9\lib\site-packages\pandas\core\internals.py", line 3681, in _multi_blockify

    list(tup_block), dtype)

 

  File "D:\WinPython-32bit-2.7.9.3\python-2.7.9\lib\site-packages\pandas\core\internals.py", line 3725, in _stack_arrays

    stacked = np.empty(shape, dtype=dtype)

 

MemoryError

</code>

使用pandas.DataFrame读175MB数据(111W行 X 38列),程序执行过程中报如上错误。

巴扎黑巴扎黑2809 days ago876

reply all(1)I'll reply

  • 巴扎黑

    巴扎黑2017-04-17 14:58:10

    Is it a reading error or a calculation error?
    If it's the latter, you can try delping some useless col after reading it, and then calculate
    I have tried it with millions of rows, but not with so many columns

    reply
    0
  • Cancelreply