this tutorial explains how to evenly divide a large go slice (e.g., 2.1m log strings) into smaller sub-slices using idiomatic, zero-allocation slicing—avoiding manual copying and off-by-one errors.
this tutorial explains how to evenly divide a large go slice (e.g., 2.1m log strings) into smaller sub-slices using idiomatic, zero-allocation slicing—avoiding manual copying and off-by-one errors.
In Go, slicing is a lightweight, reference-based operation—there’s no need to manually copy elements with loops or append. The key insight is that logs[i:j] creates a new slice header pointing into the same underlying array, making chunking both memory-efficient and fast.
To distribute ~2.1 million log strings as evenly as possible across available logical CPUs (or any desired number of chunks), use ceiling division to compute the target chunk size:
numCPU := runtime.NumCPU() chunkSize := (len(logs) + numCPU - 1) / numCPU // Equivalent to ceil(len(logs) / numCPU)
Then iterate through the original slice by stride, slicing each chunk on-the-fly:
var divided [][]string
for i := 0; i len(logs) {
end = len(logs)
}
divided = append(divided, logs[i:end])
}
✅ Why this works:
- No unnecessary allocations: each logs[i:end] reuses the original backing array.
- Handles uneven division gracefully: the last chunk naturally absorbs the remainder.
- Eliminates off-by-one bugs (e.g., i == NumCPU check in the original was unreachable and logically flawed).
⚠️ Important notes:
- Avoid pre-allocating divided with make([][]string, 0) and a capacity guess unless you’re certain about the final count—Go’s slice growth is efficient, but overestimating wastes memory. If you prefer pre-allocation: divided := make([][]string, 0, (len(logs)+chunkSize-1)/chunkSize).
- Never assume len(logs) >= numCPU: if the slice is smaller than the number of chunks, some resulting sub-slices will be empty—add a guard like if chunkSize == 0 { chunkSize = 1 } for robustness.
- This pattern applies universally—not just to []string, but to any slice type.
With this approach, your 2.1M-log slice is split in O(n) time and near-optimal memory usage—ready for parallel processing with runtime.Parallel() or goroutines.
