使用python的pandas模块时，查找和修改dataFrame中的值速度非常慢，请问是什么原因，有什么好办法解决吗？

Question

最近在用pandas做一个机器学习的项目，训练集大概2G。我用的dataFrame来操作数据，对训练集做了一次groupby和mean的操作，速度还挺快的，但把得到的结果赋值给用户参数（也是一个dataframe表）的时候，速度缺特别慢，请问这是什么原因呢？
训练集大概有7000多万行，做groupby和mean操作大概十几分钟也就完成了，但赋值操作缺超级慢，每秒钟大概只能赋值50条左右，差距太大了。其中赋值语句大概是这样操作的：
dataframeA.loc[user,'']=dataframeB.loc[user,'']。两张dataFrame表都很大，都是百万级的，不知道是不是和量级较大有关，但千万级的训练集做groupby也没什么压力啊，而且这个机器学习项目，会经常使用和修改表中的参数，如果都这么慢的话，可能就无法正常使用了，哪位大神有什么好的解释或者建议吗？非常感谢了。部分代码如下，有什么细节没有说清楚的话可以问我，我再详细解释下。

def get_average_rating(self):
        self.u = log_train['Result'].mean()
        print 'u is ',self.u
        i,j = 0,0
        user_mean_rate = log_train.groupby('UserId').agg({'Result':np.mean})
        item_mean_rate = log_train.groupby('ItemId').agg({'Result':np.mean})

        #计算bias
        print 'calc bias start'
        start_time = time.time()
        for user in train_users:
            i += 1
            if i%2000==0:
                print '2000 used time',time.time()-start_time
                start_time = time.time()
            self.user_params.loc[user,'bias'] = user_mean_rate.loc[user,'Result']-self.u
        for item in train_items:
            self.item_params.loc[item,'bias'] = item_mean_rate.loc[item,'Result']-self.u

        print 'bias total used time',time.time()-start_time

Answer 1

pandas has a generator of df.iterrows() to loop through the rows of DataFrame, which is the most efficient.

For details, please see the documentation:
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.iterrows.html

Answer 2

I have never done anything of this magnitude, but my experience is that it is best not to operate df one by one, as it is basically slow, and the entire column operation is much faster
1. Append
It is best to write all the new values ​​​​in an empty df , and then merge
But sometimes it is inevitable to append directly
2. It is faster to delete
directly using the del statement
3. Change
also adopts the merge idea, overwriting the original value

Answer 3

I don’t think the assignment is slow
self.user_params.loc[user,'bias'] is equivalent to taking the second-level index from the first-level index, which should be very slow
Can the item and user be divided into two dataframes?

Answer 4

loc is the slowest. Try to use ix instead. It's best to use iterrows to construct a loop.