How to Efficiently Decode Large Streaming JSON in Go?

How to Decode Streaming JSON in Go

When working with large JSON responses, it's not ideal to load the entire response into memory before decoding it. Using the ioutil.ReadAll function can lead to memory issues when dealing with large JSON payloads. This article will explore how to decode JSON data on the fly as it streams in, avoiding memory consumption problems.

JSON Streaming with json.Decoder

The json.Decoder in the Go standard library provides the ability to parse JSON streams incrementally. This is achieved through the Decoder.Token() method.

The Decoder.Token() method returns the next token in the JSON stream without consuming it. This allows for selective parsing of JSON data and event-driven processing.

Handling JSON Structures

Event-driven parsing requires a state machine to track the current position within the JSON structure. We can use this state machine to process different parts of the JSON data as they appear in the stream.

For example, let's say we receive a JSON response with the following format:

{
    "property1": "value1",
    "property2": "value2",
    "array": [
        { "item1": "value3" },
        { "item2": "value4" }
    ]
}

We can write a function that incrementally parses this JSON stream and processes the array element separately:

func processJSONStream(stream io.Reader) {
    decoder := json.NewDecoder(stream)

    state := "start"
    for decoder.More() {
        token, err := decoder.Token()
        if err != nil {
            log.Fatal(err)
        }

        switch state {
        case "start":
            if delim, ok := token.(json.Delim); ok && delim == '{' {
                state = "object"
            } else {
                log.Fatal("Expected object")
            }
        case "object":
            switch t := token.(type) {
            case json.Delim:
                if t == '}' {
                    // End of object
                    state = "end"
                } else if t == ',' {
                    // Next property
                    continue
                } else if t == '[' {
                    // Array found
                    state = "array"
                }

                if t == ':' {
                    // Property value expected
                    state = "prop_value"
                }
            case string:
                // Property name
                fmt.Printf("Property '%s'\n", t)
            default:
                // Property value
                fmt.Printf("Value: %v\n", t)
            }
        case "array":
            if delim, ok := token.(json.Delim); ok && delim == ']' {
                // End of array
                state = "object"
            } else if token == json.Delim('{') {
                // Array item object
                fmt.Printf("Item:\n")
                state = "item"
            }
        case "item":
            switch t := token.(type) {
            case json.Delim:
                if t == '}' {
                    // End of item object
                    fmt.Printf("\n")
                    state = "array"
                } else if t == ',' {
                    // Next item property
                    fmt.Printf(",\n")
                    continue
                }
            case string:
                // Item property name
                fmt.Printf("\t'%s'", t)
            default:
                // Item property value
                fmt.Printf(": %v", t)
            }
        case "prop_value":
            // Decode the property value
            var value interface{}
            if err := decoder.Decode(&value); err != nil {
                log.Fatal(err)
            }
            fmt.Printf("Value: %v\n", value)
            state = "object"
        }
    }
}

When called with the JSON response, this function will print the property names and values, as well as the individual items within the array.

Conclusion

Using the json.Decoder and Decoder.Token() in event-driven processing allows us to parse large JSON responses incrementally, avoiding memory consumption issues and enabling efficient processing of data as it streams in.

The above is the detailed content of How to Efficiently Decode Large Streaming JSON in Go?. For more information, please follow other related articles on the PHP Chinese website!