How Can I Unmarshal Unknown Protobuf Messages in Go?

Pitfalls of Unmarshaling Unknown Protobuf Messages

The inability of proto.Unmarshal to handle interface{} types is an essential aspect to understand. Its method signature stipulates passing a proto.Message argument, which is implemented by concrete protobuffer types.

Overcoming the Challenge

When dealing with raw protobuffer payloads that lack additional context, it's crucial to have at least some identifying information (e.g., a string or number) alongside the byte slice. This information can be used to map to the specific protobuffer concrete message. You can then use a switch statement to instantiate the appropriate type and pass it to Unmarshal:

switch atLeastSomething {
    case "foo":
        message = &mypb.Foo{}
    case "bar":
        message = &mypb.Bar{}
}
_ = proto.Unmarshal(data, message)

Embracing the Unpredictable: Decoding Completely Unknown Messages

In rare instances, you may encounter a completely unknown protobuffer payload. The protowire package provides a solution to extract some information from it, albeit with inherent limitations. The payload content can be retrieved, but with weaker semantics.

Implementation Details

Here's a simplified parser for unknown proto messages:

func parseUnknown(b []byte) []Field {
    // Data model
    type Field struct {
        Tag Tag
        Val Val
    }
    type Tag struct {
        Num int32
        Type protowire.Type
    }
    type Val struct {
        Payload interface{}
        Length  int
    }
    // Parsing algorithm
    fields := make([]Field, 0)
    for len(b) > 0 {
        n, t, fieldlen := protowire.ConsumeField(b)
        if fieldlen <p><strong>Sample Input and Output</strong></p><p>With the sample input:</p><pre class="brush:php;toolbar:false">message Foo {
  string a = 1;
  string b = 2;
  Bar bar = 3;
}
message Bar {
  string c = 1;
}
&test.Foo{A: "A", B: "B", Bar: &test.Bar{C: "C"}}

The output will be:

Field{Tag:Tag{Num:1, Type:2}, Val:Val{Payload:[]uint8{0x41}, Length:1}}
Field{Tag:Tag{Num:2, Type:2}, Val:Val{Payload:[]uint8{0x42}, Length:1}}
Field{Tag:Tag{Num:1, Type:2}, Val:Val{Payload:[]uint8{0x43}, Length:1}}
Field{Tag:Tag{Num:3, Type:2}, Val:Val{Payload:[]Field{Field{Tag:Tag{Num:1, Type:2}, Val:Val{Payload:[]uint8{0x43}, Length:1}}}, Length:3}}

Additional Considerations

• Sub-messages are recursively parsed and stored, if valid. Otherwise, bytes are stored as-is.
• Repeated fields are not explicitly handled, but can be inferred from repeated Tag.Num values post-parsing.
• Maps are expected to be semantically equivalent to repeated key/value pairs.
• Oneof information is likely lost, preserving only the set value.

Alternative: Any and Interface{}

While Any might initially seem like a suitable option for unknown messages, it has a strictly defined structure and cannot be used for decoding unknown payloads. Furthermore, protowire is not applicable to Any due to its specific structure.

The above is the detailed content of How Can I Unmarshal Unknown Protobuf Messages in Go?. For more information, please follow other related articles on the PHP Chinese website!