Reading Non-UTF-8 Text Files in Go
Reading and writing non-UTF-8 text files can be challenging in Go since the standard library assumes UTF-8 encoding. This article addresses this issue and provides a comprehensive solution using Go's sub-repositories.
Problem:
How can we read text files encoded in non-UTF-8 formats, such as GBK, in Go?
Solution:
To read files in non-UTF-8 encodings, we utilize the golang.org/x/text/encoding package. This package defines an interface for generic character encodings that facilitate conversion to and from UTF-8.
In particular, for GBK encoding, we employ the golang.org/x/text/encoding/simplifiedchinese sub-package, which provides GB18030, GBK, and HZ-GB2312 encoding implementations. These implementations implement the encoding.Encoding interface.
Implementation:
Here is an example demonstrating the reading and writing of a GBK-encoded file:
package main import ( "bufio" "fmt" "log" "os" "golang.org/x/text/encoding/simplifiedchinese" "golang.org/x/text/transform" ) var enc = simplifiedchinese.GBK func main() { // Example filename const filename = "example_GBK_file" exampleWriteGBK(filename) exampleReadGBK(filename) } func exampleReadGBK(filename string) { f, err := os.Open(filename) if err != nil { log.Fatal(err) } // Convert GBK to UTF-8 on the fly r := transform.NewReader(f, enc.NewDecoder()) sc := bufio.NewScanner(r) for sc.Scan() { fmt.Printf("Read line: %s\n", sc.Bytes()) } if err := sc.Err(); err != nil { log.Fatal(err) } } func exampleWriteGBK(filename string) { f, err := os.Create(filename) if err != nil { log.Fatal(err) } w := transform.NewWriter(f, enc.NewEncoder()) // Example text with Chinese characters _, err = fmt.Fprintln(w, `In 1995, China National Information Technology Standardization Technical Committee set down the Chinese Internal Code Specification (Chinese: 汉字内码扩展规范(GBK); pinyin: Hànzì Nèimǎ Kuòzhǎn Guīfàn (GBK)), Version 1.0, known as GBK 1.0, which is a slight extension of Codepage 936. The newly added 95 characters were not found in GB 13000.1-1993, and were provisionally assigned Unicode PUA code points.`) if err != nil { log.Fatal(err) } }
Playground:
https://go.dev/play/p/fFIy9VES6cL
The above is the detailed content of How to Read Non-UTF-8 Encoded Text Files (e.g., GBK) in Go?. For more information, please follow other related articles on the PHP Chinese website!

This article explains Go's package import mechanisms: named imports (e.g., import "fmt") and blank imports (e.g., import _ "fmt"). Named imports make package contents accessible, while blank imports only execute t

This article details efficient conversion of MySQL query results into Go struct slices. It emphasizes using database/sql's Scan method for optimal performance, avoiding manual parsing. Best practices for struct field mapping using db tags and robus

This article explains Beego's NewFlash() function for inter-page data transfer in web applications. It focuses on using NewFlash() to display temporary messages (success, error, warning) between controllers, leveraging the session mechanism. Limita

This article explores Go's custom type constraints for generics. It details how interfaces define minimum type requirements for generic functions, improving type safety and code reusability. The article also discusses limitations and best practices

This article demonstrates creating mocks and stubs in Go for unit testing. It emphasizes using interfaces, provides examples of mock implementations, and discusses best practices like keeping mocks focused and using assertion libraries. The articl

This article details efficient file writing in Go, comparing os.WriteFile (suitable for small files) with os.OpenFile and buffered writes (optimal for large files). It emphasizes robust error handling, using defer, and checking for specific errors.

The article discusses writing unit tests in Go, covering best practices, mocking techniques, and tools for efficient test management.

This article explores using tracing tools to analyze Go application execution flow. It discusses manual and automatic instrumentation techniques, comparing tools like Jaeger, Zipkin, and OpenTelemetry, and highlighting effective data visualization


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

SublimeText3 Chinese version
Chinese version, very easy to use

SublimeText3 Mac version
God-level code editing software (SublimeText3)

MantisBT
Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

Dreamweaver CS6
Visual web development tools

DVWA
Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software
