How Can I Efficiently Chunk a Large Slice of Strings in Go?

Slice Chunking in Go

You want to divide a slice of approximately 2.1 million strings into evenly distributed chunks. The code segment provided utilizes a technique that duplicates the first chunk instead of distributing the strings evenly.

The solution lies in appending slices of logs to the divided slice instead of creating new slices. By calculating the chunk size based on the number of CPUs and the slice length, the logs can be evenly distributed over chunks.

The modified code:

var divided [][]string

chunkSize := (len(logs) + numCPU - 1) / numCPU

for i := 0; i < len(logs); i += chunkSize {
    end := i + chunkSize

    if end > len(logs) {
        end = len(logs)
    }

    divided = append(divided, logs[i:end])
}

This approach ensures that the strings are distributed as evenly as possible across the chunks, maximizing performance and utilization of system resources.

The above is the detailed content of How Can I Efficiently Chunk a Large Slice of Strings in Go?. For more information, please follow other related articles on the PHP Chinese website!