Python counts the number of times a word appears_python

Recently, my manager gave me a task to count the number of occurrences of each word in a file and list the five most frequently occurring words. This article brings you an analysis of the idea of ​​counting the number of words in python. Friends who need it can refer to it

Title:

Counting in a file The number of occurrences of each word, list the 5 most frequently occurring words.

Foreword:

This question is widely used in practical application scenarios, such as the statistics of high-level students who have appeared in the CET-4 and CET-6 exams over the years. Frequency vocabulary, I remember that Li Xiaolai used his programming skills to publish a best-selling book on word memorization. He memorized words based on word frequency, which was very popular among students. This is a typical scenario where programming skills are used to solve real problems. In addition, during data analysis, those word cloud effects are essentially based on word frequency statistics to adjust the font size. If you can skillfully use the knowledge in Python to solve problems, it means that you are really getting started with Python.

Analysis

This question mainly examines the following knowledge points:

1. How to read correctly Write files

To read and write files in python, you can use the built-in function open(), and the open function has certain differences in python2 and python3. For example, in Python, you can specify the encoding format for reading and writing files. This is not possible with Python. In order to be compatible with both 2 and 3, we usually use the open function under the io module. You can check the documentation to find out the difference between them, and cultivate active learning capabilities and the habit of checking information.

Another point is that the file descriptor needs to be closed after reading and writing the file. In addition to using the try...except...finally syntax, we can also use the more elegant with... as syntax. to automatically close the file.

2. How to sort data

The sorted function is a frequently used built-in function, and its usage is also very powerful because it can specify parameters key to perform custom sorting, which means that you can not only sort numbers and letters, but also sort lists, dictionaries, and custom objects. You only need to tell the sorted function what the sorting rules are, such as For a people object, I can sort it by age or height and weight, so this function is very flexible. In addition, there is a sort method for list objects. If you can clearly distinguish the difference between list.sort and sorted That means you can already use it flexibly.

3. Use of dictionary data type

To do word frequency statistics, using a dictionary is undoubtedly the most appropriate data type. Words are used as the keys of the dictionary, and the number of times a word appears is used as The value of the dictionary conveniently records the frequency of each word. The dictionary is much like our phone book, with each name associated with a phone number. In addition, the biggest feature of the dictionary is that its query speed is very fast. Ideally, the time complexity is O(1). I mean ideally. If you want to learn more about dictionaries, I recommend reading this article https://www.laurentluce.com/posts/python-dictionary-implementation/

4. Application of regular expressions

For text and string processing, regular expressions are simply an artifact. They are widely used whether it is for data crawling or data cleaning. , of course, regular expressions are not unique to Python, they are supported by all programming languages. What we have to do is not only learn regular expressions but also its API. Only when we are familiar with the API can we apply it to actual scenarios. I recommend an article about regular expressions: http://www.cnblogs.com/huxi/archive/2010/07/04/1771073.html. In addition, I also found that some students introduced the jieba word segmentation library. This library is doing Chinese word segmentation is very useful. If you are interested, you can learn about it.

Implementation

After the analysis, we can actually implement it very quickly. So when we get a requirement, we must first clarify the requirement and think about what technologies can be used to achieve it, and then start writing code. In fact, at work, we actually spend less than half of the time writing code. .

# -*- coding:utf-8 -*-
import io
import re
class Counter:
 def __init__(self, path):
 """
 :param path: 文件路径
 """
 self.mapping = dict()
 with io.open(path, encoding="utf-8") as f:
  data = f.read()
  words = [s.lower() for s in re.findall("\w+", data)]
  for word in words:
  self.mapping[word] = self.mapping.get(word, 0) + 1
 def most_common(self, n):
 assert n > 0, "n should be large than 0"
 return sorted(self.mapping.items(), key=lambda item: item[1], reverse=True)[:n]
if __name__ == '__main__':
 most_common_5 = Counter("importthis.txt").most_common(5)
 for item in most_common_5:
 print(item)

Print result:

('is', 10)
('better', 8)
('than', 8)
('the', 6)
('to', 5)

Summary

When I look at your code, many codes still have irregular naming (it is recommended to read PEP8), and the code layout is confusing (it is difficult to read, it is recommended to use Pycharm for formatting). There are also many codes whose implementation methods look very complicated (often the more complex the code, the more bugs it has). Of course, the implementation method is not the only one.

For example, the Python module itself provides a collections.Counter class, which inherits from the dict class and is used for statistics. I found that some students use this class to implement it. If you are careful, you may find it. , the Counter I implemented is very similar to the Counter under collections. In fact, this is wheel-making. Wheel-making can exercise our programming thinking. Of course, if there are ready-made things at work, there is no need to make wheels yourself, unless you have the confidence to do it. Better. You can also think about what you would do if Python did not provide the Counter tool.

In addition, this module also provides an ordered dictionary object OrderedDict, which can save us from manual sorting operations. Finally, I recommend that you study and summarize all the content I mentioned above. If you can persist for 100 days, I believe you will have a good grasp of Python.

Related recommendations:

Python implements two-dimensional array output as a picture_python

Python implements typing instance attributes examine

The above is the detailed content of Python counts the number of times a word appears_python. For more information, please follow other related articles on the PHP Chinese website!