How to Efficiently Remove Diacritics from UTF-8 Strings in Go?

Removing Diacritics in Go

When working with UTF8 encoded strings, it may be necessary to remove diacritics, such as the accents from "žůžo" to get "zuzo". To handle such scenarios efficiently, there are standard libraries and techniques available in Go.

One approach involves leveraging the unicode.Is() function to identify diacritics (characters classified as "Mn" for nonspacing marks).

The following code snippet demonstrates how to remove diacritics from a given string utilizing the unicode/norm and golang.org/x/text/transform packages:

package main

import (
    "fmt"
    "unicode"

    "golang.org/x/text/transform"
    "golang.org/x/text/unicode/norm"
)

func isMn(r rune) bool {
    return unicode.Is(unicode.Mn, r) // Mn: nonspacing marks
}

func main() {
    t := transform.Chain(norm.NFD, transform.RemoveFunc(isMn), norm.NFC)
    result, _, _ := transform.String(t, "žůžo")
    fmt.Println(result)
}

This code removes diacritics by applying a series of transformations:

1. Normalized Form Decomposition (NFD): Breaks down the string into its base Unicode characters, including diacritics.
2. RemoveFunc(isMn): Filters out characters that are nonspacing marks (diacritics).
3. Normalization Form Composition (NFC): Recomposes the string without diacritics.

As a result, the output will be a string stripped of diacritics, as in the example: "žůžo" => "zuzo".

The above is the detailed content of How to Efficiently Remove Diacritics from UTF-8 Strings in Go?. For more information, please follow other related articles on the PHP Chinese website!