Home > Article > Backend Development > NLTK learning: classifying and annotating vocabulary
[TOC]
Many subsequent tasks require tagged words. nltk comes with its own English tagger
pos_tag
import nltk text = nltk.word_tokenize("And now for something compleyely difference")print(text)print(nltk.pos_tag(text))
nltk.tag.str2tuple('word/type')
text = "The/AT grand/JJ is/VBD ."print([nltk.tag.str2tuple(t) for t in text.split()])
nltk corpus ue navel provides unification interface, you can ignore different file formats. Format:
Corpus.tagged_word()/tagged_sents()
. Parameters can specify categories and fields
print(nltk.corpus.brown.tagged_words())
Here we take nouns as an example
from nltk.corpus import brown word_tag = nltk.FreqDist(brown.tagged_words(categories="news"))print([word+'/'+tag for (word,tag)in word_tag if tag.startswith('V')])################下面是查找money的不同标注#################################wsj = brown.tagged_words(categories="news") cfd = nltk.ConditionalFreqDist(wsj)print(cfd['money'].keys())
def findtag(tag_prefix,tagged_text): cfd = nltk.ConditionalFreqDist((tag,word) for (word,tag) in tagged_text if tag.startswith(tag_prefix))return dict((tag,list(cfd[tag].keys())[:5]) for tag in cfd.conditions())#数据类型必须转换为list才能进行切片操作tagdict = findtag('NN',nltk.corpus.brown.tagged_words(categories="news"))for tag in sorted(tagdict):print(tag,tagdict[tag])
Required
nltk.bigrams()
andnltk.trigrams()
correspond to the 2-gram model and the 3-gram model respectively.
brown_tagged = brown.tagged_words(categories="learned") tags = [b[1] for (a,b) in nltk.bigrams(brown_tagged) if a[0]=="often"] fd = nltk.FreqDist(tags) fd.tabulate()
The simplest tagger is for each identifier Assign uniform tags. Below is a tagger that turns all words into NN. And use
evaluate()
to test. It facilitates first analysis and improves stability when many words are nouns.
brown_tagged_sents = brown.tagged_sents(categories="news") raw = 'I do not like eggs and ham, I do not like them Sam I am'tokens = nltk.word_tokenize(raw) default_tagger = nltk.DefaultTagger('NN')#创建标注器print(default_tagger.tag(tokens)) # 调用tag()方法进行标注print(default_tagger.evaluate(brown_tagged_sents))
Note that the rules here are fixed (determined by yourself). As the rules become more and more complete, the accuracy becomes higher.
patterns = [ (r'.*ing$','VBG'), (r'.*ed$','VBD'), (r'.*es$','VBZ'), (r'.*','NN')#为了方便,只有少量规则] regexp_tagger = nltk.RegexpTagger(patterns) regexp_tagger.evaluate(brown_tagged_sents)
There is a difference between this and the book. It is different from python2. Pay attention to debugging. The query tagger stores the most likely tags, and the
backoff
parameter can be set. If the tag cannot be marked, use this tagger (this process is backoff)
fd = nltk.FreqDist(brown.words(categories="news")) cfd = nltk.ConditionalFreqDist(brown.tagged_words(categories="news"))##############################################python2和3的区别#########most_freq_words = fd.most_common(100) likely_tags = dict((word,cfd[word].max()) for (word,times) in most_freq_words)#######################################################################baseline_tagger = nltk.UnigramTagger(model=likely_tags,backoff=nltk.DefaultTagger('NN')) baseline_tagger.evaluate(brown_tagged_sents)
The behavior of the unary annotator and the search annotator Very similar to the techniques used to build unary annotators, train for .
Here our annotator only memorizes the training set instead of building a general model. The agreement is very good, but it cannot be generalized to new texts.
size = int(len(brown_tagged_sents)*0.9) train_sents = brown_tagged_sents[:size] test_sents = brown_tagged_sents[size+1:] unigram_tagger = nltk.UnigramTagger(train_sents) unigram_tagger.evaluate(test_sents)
N-gram tagger is to retrieve the word with index= n, and retrieve n-N< ;=index<=n-1 tag. That is, the tag of the current word is further determined through the tag tag of the previous word. Similar to
nltk.UnigramTagger()
, the built-in binary tagger is:nltk.BigramTagger()
The usage is consistent.
Many times, an algorithm with wider coverage is more useful than an algorithm with higher accuracy. Use
backoff
to specify the backoff annotator to implement the combination of annotators. If the parametercutoff
is explicitly declared as int type, contexts that only appear 1-n times will be automatically discarded.
t0 = nltk.DefaultTagger('NN') t1 = nltk.UnigramTagger(train_sents,backoff=t0) t2 = nltk.BigramTagger(train_sents,backoff=t1) t2.evaluate(test_sents)
It can be found that after comparing with the original, the accuracy is significantly improved
For the beginning of the sentence of words, there are no first n words. Solution: Train the tagger with tagged tagged_sents.
is superior to the above. The idea of implementation: start with a big stroke, then fix the details, and make detailed changes little by little.
Not only does it take up a small amount of memory, but it is also associated with the context, and corrects errors in real time as the problem becomes smaller, rather than static. Of course, the calls are different in python3 and python2.
from nltk.tag import brill brill.nltkdemo18plus() brill.nltkdemo18()
The above is the detailed content of NLTK learning: classifying and annotating vocabulary. For more information, please follow other related articles on the PHP Chinese website!