['000001_2017-03-17.csv', '000001_2017-03-20.csv',
'000002_2017-03-21.csv', '000002_2017-03-22.csv',
'000003_2017-03-23.csv', '000004_2017-03-24.csv']
numpy數組,總共有幾個萬個元素。現在要保留每個元素前面的編號000001之類的,並且去掉重複,只保留唯一的一個編號。結果應該是['000001','000002','000003','000004']
除了用for語句實現外,有沒有更有效率的辦法?
迷茫2017-06-30 09:58:09
寫個NumPy的吧~
python3
>>> import numpy as np
>>> a = np.array(['000001_2017-03-17.csv', '000001_2017-03-20.csv',
'000002_2017-03-21.csv', '000002_2017-03-22.csv',
'000003_2017-03-23.csv', '000004_2017-03-24.csv'])
>>> b = np.unique(np.fromiter(map(lambda x:x.split('_')[0],a),'|S6'))
>>> b
array([b'000001', b'000002', b'000003', b'000004'],
dtype='|S6')
還可以這樣寫:np.frompyfunc
'|S6'
是以6個位元組儲存字串
'
小端序Unicode字元
儲存字串
>>> b = np.array(np.unique(np.frompyfunc(lambda x:x[:6],1,1)(a)),dtype='<U6')
>>> b
array(['000001', '000002', '000003', '000004'],
dtype='<U6')
学习ing2017-06-30 09:58:09
綜合兩位仁兄的寫法
@同意並接受 @xiaojieluoff
如果編號長度固定是前六位,最快的寫法下面第一種最快
import time
lst = ['000001_2017-03-17.csv', '000001_2017-03-20.csv', '000002_2017-03-21.csv', '000002_2017-03-22.csv', '000003_2017-03-23.csv', '000004_2017-03-24.csv'] * 1000000
start = time.time()
data = {_[:6] for _ in lst}
print 'dic: {}'.format(time.time() - start)
start = time.time()
data = set(_[:6] for _ in lst)
print 'set: {}'.format(time.time() - start)
start = time.time()
data = set(map(lambda _: _[:6], lst))
print('map:{}'.format(time.time() - start))
start = time.time()
data = set()
[data.add(_[:6]) for _ in lst]
print('for:{}'.format(time.time() - start))
耗时:
dic: 0.72798705101
set: 0.929664850235
map:1.89214396477
for:1.76194214821
某草草2017-06-30 09:58:09
使用 map 和匿名函數
lists = ['000001_2017-03-17.csv', '000001_2017-03-20.csv','000002_2017-03-21.csv','000002_2017-03-22.csv','000003_2017-03-23.csv', '000004_2017-03-24.csv']
data = list(set(map(lambda x:x.split('_')[0], lists)))
print(data)
輸出:
['000003', '000004', '000001', '000002']
運行下面程式碼可以看到 , 在 6百萬 條資料下,map 比 for 快了 0.6s 左右
import time
lists = ['000001_2017-03-17.csv', '000001_2017-03-20.csv', '000002_2017-03-21.csv', '000002_2017-03-22.csv', '000003_2017-03-23.csv', '000004_2017-03-24.csv'] * 1000000
map_start = time.clock()
map_data = list(set(map(lambda x:x.split('_')[0], lists)))
map_end = (time.clock() - map_start)
print('map 运行时间:{}'.format(map_end))
for_start = time.clock()
data = set()
for k in lists:
data.add(k.split('_')[0])
for_end = (time.clock() - for_start)
print('for 运行时间:{}'.format(for_end))
輸出:
map 运行时间:2.36173
for 运行时间:2.9405870000000003
如果把測試數據擴大到 6千萬, 差距就更明顯了
map 运行时间:29.620203
for 运行时间:33.132621