php小编小新在这里为大家介绍一种非常有趣的技术——Participle expresses unexpected mark。分词是自然语言处理中的一个重要任务,它将一段文本按照一定的规则进行切分,得到一系列的词语。而在分词过程中,我们有时会遇到一些特殊的情况,比如在某些词语中出现了意外的标记,这可能会对后续的处理造成困扰。因此,研究如何表示和处理这种意外的标记就变得非常重要。在本文中,我们将介绍一些常见的意外标记,并给出相应的解决方案,希望能对大家有所帮助。
问题内容
我正在玩一个分词来学习如何解析,但我无法确定为什么这是意外的。
// nolint: golint, dupl package main import ( "fmt" "io" "github.com/alecthomas/participle/v2" "github.com/alecthomas/participle/v2/lexer" ) var htaccesslexer = lexer.mustsimple([]lexer.simplerule{ {"comment", `^#[^\n]*`}, {"ident", `^\w+`}, {"int", `\d+`}, {"string", `("(\\"|[^"])*"|\s+)`}, {"eol", `[\n\r]+`}, {"whitespace", `[ \t]+`}, }) type htaccess struct { directives []*directive `@@*` } type directive struct { pos lexer.position errordocument *errordocument `@@` } type errordocument struct { code int `"errordocument" @int` path string `@string` } var htaccessparser = participle.mustbuild[htaccess]( participle.lexer(htaccesslexer), participle.caseinsensitive("ident"), participle.unquote("string"), participle.elide("whitespace"), ) func parse(r io.reader) (*htaccess, error) { program, err := htaccessparser.parse("", r) if err != nil { return nil, err } return program, nil } func main() { v, err := htaccessparser.parsestring("", `errordocument 403 test`) if err != nil { panic(err) } fmt.println(v) }
据我所知,这似乎是正确的,我期望 403 在那里,但我不确定为什么它不识别它。
编辑: 我将词法分析器更改为:
var htaccessLexer = lexer.MustSimple([]lexer.SimpleRule{ {"dir", `^\w+`}, {"int", `\d+`}, {"str", `("(\\"|[^"])*"|\S+)`}, {"EOL", `[\n\r]+`}, {"whitespace", `\s+`}, })
错误消失了,但它仍然打印一个空数组,不知道为什么。我也不确定为什么对词法分析器使用不同的值可以修复它。
解决方法
我相信我发现了问题,这是顺序,ident 通过 \w 标签在我的词法分析器中查找数字,因此这导致我的整数被标记为 ident。
我发现我必须将 quotedstrings 和 unquotedstrings 分开,否则未加引号的字符串会获取整数。或者,我可以确保它只获取非数字值,但这会错过 stringwithnum2
之类的东西
这是我的解决方案
var htaccesslexer = lexer.mustsimple([]lexer.simplerule{ {"comment", `(?i)#[^\n]*`}, {"quotedstring", `"(\\"|[^"])*"`}, {"number", `[-+]?(\d*\.)?\d+`}, {"unquotedstring", `[^ \t]+`}, {"ident", `^[a-za-z_]`}, {"eol", `[\n\r]+`}, {"whitespace", `[ \t]+`}, })
type ErrorDocument struct { Pos lexer.Position Code int `"ErrorDocument" @Number` Path string `(@QuotedString | @UnQuotedString)` }
这解决了我的问题,因为它现在查找带引号的字符串,然后查找数字,然后查找不带引号的字符串。
The above is the detailed content of Participle expresses unexpected mark. For more information, please follow other related articles on the PHP Chinese website!

This article explains Go's package import mechanisms: named imports (e.g., import "fmt") and blank imports (e.g., import _ "fmt"). Named imports make package contents accessible, while blank imports only execute t

This article explains Beego's NewFlash() function for inter-page data transfer in web applications. It focuses on using NewFlash() to display temporary messages (success, error, warning) between controllers, leveraging the session mechanism. Limita

This article details efficient conversion of MySQL query results into Go struct slices. It emphasizes using database/sql's Scan method for optimal performance, avoiding manual parsing. Best practices for struct field mapping using db tags and robus

This article explores Go's custom type constraints for generics. It details how interfaces define minimum type requirements for generic functions, improving type safety and code reusability. The article also discusses limitations and best practices

This article demonstrates creating mocks and stubs in Go for unit testing. It emphasizes using interfaces, provides examples of mock implementations, and discusses best practices like keeping mocks focused and using assertion libraries. The articl

This article details efficient file writing in Go, comparing os.WriteFile (suitable for small files) with os.OpenFile and buffered writes (optimal for large files). It emphasizes robust error handling, using defer, and checking for specific errors.

The article discusses writing unit tests in Go, covering best practices, mocking techniques, and tools for efficient test management.

This article explores using tracing tools to analyze Go application execution flow. It discusses manual and automatic instrumentation techniques, comparing tools like Jaeger, Zipkin, and OpenTelemetry, and highlighting effective data visualization


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

WebStorm Mac version
Useful JavaScript development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

SublimeText3 Chinese version
Chinese version, very easy to use

Safe Exam Browser
Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

Dreamweaver Mac version
Visual web development tools
