Share some tips on python data statistics

I have recently been using Python to do data statistics. Here are some tips that I have found and summarized recently. I hope it can help some children in this area. Some techniques are very common usage, and we usually don’t pay attention to them, but in specific scenarios, these small methods can still bring great help.

1. Map keys to multiple values ​​in the dictionary

{'b': [4, 5, 6], 
'a': [1, 2, 3]}

Sometimes when we count the same key values, we want to add all entries with the same key to a dictionary with key as the key, and then perform various operations. At this time, we can use the following code to operate:

from collections import defaultdict
d = defaultdict(list)
print(d)
d['a'].append(1)
d['a'].append(2)
d['a'].append(3)
d['b'].append(4)
d['b'].append(5)
d['b'].append(6)
print(d)
print(d.get("a"))
print(d.keys())
print([d.get(i) for i in d])

The methods in collections are used here. There are many useful methods in there. We have time to continue to understand them in depth.

The result of running the above code:

defaultdict(, {})
defaultdict(, {'b': [4, 5, 6], 'a': [1, 2, 3]})
[1, 2, 3]
dict_keys(['b', 'a'])
[[4, 5, 6], [1, 2, 3]]

After we fill in the data, it is equivalent to quickly grouping, and then traverse each group to count some of the data we need.

2. Quickly convert dictionary key-value pairs

data = {...}
zip(data.values(), data.keys())

data is our format data. Use zip for fast key-value conversion, and then you can use functions such as max and min for data operations.

3. Sort dictionary by common key

from operator import itemgetter
data = [
  {'name': "bran", "uid": 101},
  {'name': "xisi", "uid": 102},
  {'name': "land", "uid": 103}
]
print(sorted(data, key=itemgetter("name")))
print(sorted(data, key=itemgetter("uid")))

The data format is data. If we want to sort names or uids, we use the method in the code.
Running result:

[{'name': 'bran', 'uid': 101}, {'name': 'land', 'uid': 103}, {'name': 'xisi', 'uid': 102}]
[{'name': 'bran', 'uid': 101}, {'name': 'xisi', 'uid': 102}, {'name': 'land', 'uid': 103}]

Just as we expected

4. Group multiple dictionaries in the list according to a certain field

Please note that the data must be sorted first before grouping. The sorting field is selected according to actual requirements

Data to be processed:

rows = [
  {'name': "bran", "uid": 101, "class": 13},
  {'name': "xisi", "uid": 101, "class": 11},
  {'name': "land", "uid": 103, "class": 10}
]

Expected processing results:

{
101: [{'name': 'xisi', 'class': 11, 'uid': 101},{'name': 'bran', 'class': 13, 'uid': 101}],
103: [{'name': 'land', 'class': 10, 'uid': 103}]
}

We group by uid, this is just a demonstration, uid generally will not be repeated.

This is a bit more complicated, let’s break it down step by step

some = [('a', [1, 2, 3]), ('b', [4, 5, 6])]
print(dict(some))

Result:

{'b': [4, 5, 6], 'a': [1, 2, 3]}

Our purpose here is to convert tuples into dictionaries. This is very simple and everyone should understand it. Then let’s take the next step to sort the data to be processed:

data_one = sorted(rows, key=itemgetter("class"))
print(data_one)
data_two = sorted(rows, key=lambda x: (x["uid"], x["class"]))
print(data_two)

Here we provide two sorting methods with the same principle, but the styles are slightly different. The first data_one uses itemgetter directly. As we have used before, it sorts directly according to a certain field, but sometimes we have another one. Requirements:

First sort by a certain field, and then sort by another field when the first field is repeated.

At this time, we will use the second method to sort multi-field values.
The sorting results are as follows:

[{'name': 'land', 'class': 10, 'uid': 103}, {'name': 'xisi', 'class': 11, 'uid': 101}, {'name': 'bran', 'class': 13, 'uid': 101}]
[{'name': 'xisi', 'class': 11, 'uid': 101}, {'name': 'bran', 'class': 13, 'uid': 101}, {'name': 'land', 'class': 10, 'uid': 103}]

If you take a look at the results, there are still slight differences.

Then comes the last step, combining the two methods we just talked about:

data = dict([(g, list(k)) for g, k in groupby(data_two, key=lambda x: x["uid"])])
print(data)

We group the sorted data, then generate a list of tuples, and finally convert it into a dictionary. We are done here, we have successfully grouped the data.

Some tips on python data statistics are shared here, you can refer to them if you need them.