After unzipping, take out the following files:
Training data: icwb2-data/training/pku_ training.utf8
Test data: icwb2-data/testing/pku_ test.utf8
Correct word segmentation result: icwb2-data/gold/pku_ test_ gold.utf8
Scoring tool: icwb2-data/script/socre
2 Algorithm description
The algorithm is the simplest forward maximum matching (FMM):
Generate a dictionary with training data
Pair the test data from Scan from left to right, and when you encounter the longest word, split it until the end of the sentence. Note: This is the original algorithm. This way, the code can be controlled within 60 lines. Later, looking at the test results, it was found that it was not processed well. For numerical problems, the processing of numbers has been added.
3 Source code and comments
#! /usr/bin/env python # -*- coding: utf-8 -*- # Author: minix # Date: 2013-03-20 import codecs import sys # 由规则处理的一些特殊符号 numMath = [u'0', u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9'] numMath_suffix = [u'.', u'%', u'亿', u'万', u'千', u'百', u'十', u'个'] numCn = [u'一', u'二', u'三', u'四', u'五', u'六', u'七', u'八', u'九', u'〇', u'零'] numCn_suffix_date = [u'年', u'月', u'日'] numCn_suffix_unit = [u'亿', u'万', u'千', u'百', u'十', u'个'] special_char = [u'(', u')'] def proc_num_math(line, start): """ 处理句子中出现的数学符号 """ oldstart = start while line[start] in numMath or line[start] in numMath_suffix: start = start + 1 if line[start] in numCn_suffix_date: start = start + 1 return start - oldstart def proc_num_cn(line, start): """ 处理句子中出现的中文数字 """ oldstart = start while line[start] in numCn or line[start] in numCn_suffix_unit: start = start + 1 if line[start] in numCn_suffix_date: start = start + 1 return start - oldstart def rules(line, start): """ 处理特殊规则 """ if line[start] in numMath: return proc_num_math(line, start) elif line[start] in numCn: return proc_num_cn(line, start) def genDict(path): """ 获取词典 """ f = codecs.open(path,'r','utf-8') contents = f.read() contents = contents.replace(u'\r', u'') contents = contents.replace(u'\n', u'') # 将文件内容按空格分开 mydict = contents.split(u' ') # 去除词典List中的重复 newdict = list(set(mydict)) newdict.remove(u'') # 建立词典 # key为词首字,value为以此字开始的词构成的List truedict = {} for item in newdict: if len(item)>0 and item[0] in truedict: value = truedict[item[0]] value.append(item) truedict[item[0]] = value else: truedict[item[0]] = [item] return truedict def print_unicode_list(uni_list): for item in uni_list: print item, def divideWords(mydict, sentence): """ 根据词典对句子进行分词, 使用正向匹配的算法,从左到右扫描,遇到最长的词, 就将它切下来,直到句子被分割完闭 """ ruleChar = [] ruleChar.extend(numCn) ruleChar.extend(numMath) result = [] start = 0 senlen = len(sentence) while start < senlen: curword = sentence[start] maxlen = 1 # 首先查看是否可以匹配特殊规则 if curword in numCn or curword in numMath: maxlen = rules(sentence, start) # 寻找以当前字开头的最长词 if curword in mydict: words = mydict[curword] for item in words: itemlen = len(item) if sentence[start:start+itemlen] == item and itemlen > maxlen: maxlen = itemlen result.append(sentence[start:start+maxlen]) start = start + maxlen return result def main(): args = sys.argv[1:] if len(args) < 3: print 'Usage: python dw.py dict_path test_path result_path' exit(-1) dict_path = args[0] test_path = args[1] result_path = args[2] dicts = genDict(dict_path) fr = codecs.open(test_path,'r','utf-8') test = fr.read() result = divideWords(dicts,test) fr.close() fw = codecs.open(result_path,'w','utf-8') for item in result: fw.write(item + ' ') fw.close() if __name__ == "__main__": main()
4 Testing and scoring results
Use dw.py to train the data, test the data, and generate the result file
Use score to score based on the training data, correct word segmentation results, and the results we generated
Use tail to check the overall score of the last few lines of the result file. In addition, socre.utf8 also provides a large number of comparison results, which can be used to find out where your own word segmentation results are not good enough
Note: The entire testing process is in Ubuntu Completed below
$ python dw.py pku_training.utf8 pku_test.utf8 pku_result.utf8
$ perl score pku_training.utf8 pku_test_gold.utf8 pku_result.utf8 > score.utf8
$ tail -22 score.utf8
INS ERTIONS: 0
DELETIONS: 0
SUBSTITUTIONS: 0
NCHANGE: 0
NTRUTH: 27
NTEST: 27
TRUE WORDS RECALL: 1.000
TEST WORDS PRECISION: 1.000
=== SUMMARY:
== = TOTAL INSERTIONS: 4623
=== TOTAL DELETIONS: 1740
=== TOTAL SUBSTITUTIONS: 6650
=== TOTAL NCHANGE: 13013
=== TOTAL TRUE WORD COUNT: 104372
=== TOTAL TEST WORD COUNT: 107255
=== TOTAL TRUE WORDS RECALL: 0.920
=== TOTAL TEST WORDS PRECISION: 0.895
=== F MEASURE: 0.907
=== OOV Rate: 0.940
=== OOV Recall Rate: 0.917
=== IV Recall Rate: 0.966
The FMM algorithm based on the dictionary is a very basic word segmentation algorithm. The effect is not that good, but it is simple enough and easy to start. As the learning deepens, I may also use Python to implement other word segmentation algorithms. Another feeling is that when reading, try to realize as much as possible. This will give you enough enthusiasm to pay attention to every detail of the theory, and you will not feel so boring.

Python excels in gaming and GUI development. 1) Game development uses Pygame, providing drawing, audio and other functions, which are suitable for creating 2D games. 2) GUI development can choose Tkinter or PyQt. Tkinter is simple and easy to use, PyQt has rich functions and is suitable for professional development.

Python is suitable for data science, web development and automation tasks, while C is suitable for system programming, game development and embedded systems. Python is known for its simplicity and powerful ecosystem, while C is known for its high performance and underlying control capabilities.

You can learn basic programming concepts and skills of Python within 2 hours. 1. Learn variables and data types, 2. Master control flow (conditional statements and loops), 3. Understand the definition and use of functions, 4. Quickly get started with Python programming through simple examples and code snippets.

Python is widely used in the fields of web development, data science, machine learning, automation and scripting. 1) In web development, Django and Flask frameworks simplify the development process. 2) In the fields of data science and machine learning, NumPy, Pandas, Scikit-learn and TensorFlow libraries provide strong support. 3) In terms of automation and scripting, Python is suitable for tasks such as automated testing and system management.

You can learn the basics of Python within two hours. 1. Learn variables and data types, 2. Master control structures such as if statements and loops, 3. Understand the definition and use of functions. These will help you start writing simple Python programs.

How to teach computer novice programming basics within 10 hours? If you only have 10 hours to teach computer novice some programming knowledge, what would you choose to teach...

How to avoid being detected when using FiddlerEverywhere for man-in-the-middle readings When you use FiddlerEverywhere...

Error loading Pickle file in Python 3.6 environment: ModuleNotFoundError:Nomodulenamed...


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

SublimeText3 Chinese version
Chinese version, very easy to use

mPDF
mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

DVWA
Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

Dreamweaver Mac version
Visual web development tools

SecLists
SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.