How to improve the effect of jieba word segmentation to better extract keywords in scenic spot comments?

Strategies for improving Jieba word segmentation and scenic spot comment keyword extraction

Many people use Jieba for Chinese word segmentation and combine LDA models to extract the keywords of scenic spot comments, but word segmentation often affects the accuracy of the final result. For example, if you use Jieba word segmentation directly and then perform LDA modeling, the extracted topic keywords may have word segmentation errors.

The following code example shows this problem:

 # Load the Chinese stop word stop_words = set(stopwords.words('chinese'))
broadcastVar = spark.sparkContext.broadcast(stop_words)

# Chinese text participle def tokenize(text):
    return list(jieba.cut(text))

# Delete the Chinese stop word def delete_stopwords(tokens, stop_words):
    filtered_words = [word for word in tokens if word not in stop_words]
    filtered_text = ' '.join(filtered_words)
    return filtered_text

# Remove punctuation and specific characters def remove_punctuation(input_string):
    punctuation = string.punctuation "!?｡.》#Ｅ%&＇()＊＋,－/:;＜＝＞＿｜｝］＿｟｠ｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏｏ
    translator = str.maketrans('', '', punctuation)
    no_punct = input_string.translate(translator)
    return no_punct

def Thematic_focus(text):
    from gensim import corpora, models
    num_words = min(len(text) // 50 3, 10) # Dynamically adjust the number of topic words tokens = tokenize(text)
    stop_words = broadcastVar.value
    text = delete_stopwords(tokens, stop_words)
    text = remove_punctuation(text)
    tokens = tokenize(text)

    dictionary = corporate.Dictionary([tokens])
    corpus = [dictionary.doc2bow(tokens)]
    lda_model = models.LdaModel(corpus, num_topics=1, id2word=dictionary, passes=50)
    topics = lda_model.show_topics(num_words=num_words)
    for topic in topics:
        return str(topic)

In order to improve word segmentation effect and keyword extraction, the following strategies are recommended:

1. Building a custom vocabulary: Collect professional vocabulary related to tourism, build a custom vocabulary and load it into Jieba, and improve the accuracy of recognition of terms in the tourism field. This is more effective than relying on a common thesaurus.

2. Optimize the vocabulary database of stop word: Use a more comprehensive vocabulary database, or build a custom vocabulary database based on the characteristics of scenic spot comments to remove interfering words, and improve the accuracy of the LDA model. Consider using the discontinuation vocabulary published on GitHub as the basis and add or delete it according to the actual situation.

Through the above methods, the accuracy of Jieba word segmentation can be significantly improved, thereby more effectively extracting keywords in scenic spot comments, and ultimately obtaining a more accurate theme model and word cloud map. The number of topic words has also been dynamically adjusted in the code to avoid too few or too many topic words affecting the results.

The above is the detailed content of How to improve the effect of jieba word segmentation to better extract keywords in scenic spot comments?. For more information, please follow other related articles on the PHP Chinese website!