How to Normalize Non-ASCII Text Input to ASCII Using the Go Standard Library?

Normalizing Text Input to ASCII

Many applications require the ability to parse user input and identify writing errors. However, handling non-ASCII characters, such as curly quotes, can be a challenge.

String Replacement with stdlib

One common approach is to use string replacements with functions like bytes.Replace. While effective for specific known characters, this method can be tedious for handling various characters.

Using the Strings.Map Function

The Go standard library provides a more versatile solution with the strings.Map function. This function allows for the mapping of runes (Unicode code points) to other runes. By defining a custom mapping function, you can convert non-ASCII characters to their ASCII equivalents.

Example

<code class="go">import (
    "fmt"
    "strings"
)

func normalize(in rune) rune {
    switch in {
    case '“', '‹', '”', '›':
        return '"'
    case '‘', '’':
        return '\''
    }
    return in
}

func main() {
    data := "Hello “Frank” or ‹François› as you like to be ‘called’"
    fmt.Printf("Original: %s\n", data)
    cleanedData := strings.Map(normalize, data)
    fmt.Printf("Cleaned: %s\n", cleanedData)
}</code>

Output:

Original: Hello “Frank” or ‹François› as you like to be ‘called’
Cleaned: Hello "Frank" or "François" as you like to be 'called'

The above is the detailed content of How to Normalize Non-ASCII Text Input to ASCII Using the Go Standard Library?. For more information, please follow other related articles on the PHP Chinese website!