How Can I Parse Unknown Protobuf Messages Without Type Information?

Unknown Proto Message Demarshalling Without Type Information

While protobuf's proto.Unmarshal() method requires a known message type, unknown messages present a challenge. However, we can extract limited information using the protowire package.

Approach:

1. Use protowire.ConsumeField() to iterate through the unknown message payload.

2. Identify the field type, which could be:

   • Varint
   • Fixed64
   • Bytes
   • StartGroup
   • Fixed32
3. Parse the field payload based on its type.
4. Handle sub-messages by recursively parsing their payloads.

Data Model:

type Field struct {
    Tag Tag
    Val Val
}

type Tag struct {
    Num int32
    Type protowire.Type
}

type Val struct {
    Payload interface{}
    Length int
}

Parser:

func parseUnknown(b []byte) []Field {
    // Iteratively consume and parse fields
    for len(b) > 0 {
        // Read field tag and length
        n, t, fieldlen := protowire.ConsumeField(b)
        if fieldlen <p><strong>Example (Message Structure):</strong></p><pre class="brush:php;toolbar:false">message Foo {
  string a = 1;
  string b = 2;
  Bar bar = 3;
}

message Bar {
  string c = 1;
}

Example (Parsed Output):

main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x41}, Length:1}}
main.Field{Tag:main.Tag{Num:2, Type:2}, Val:main.Val{Payload:[]uint8{0x42}, Length:1}}
main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x43}, Length:1}}
main.Field{Tag:main.Tag{Num:3, Type:2}, Val:main.Val{Payload:[]main.Field{main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x43}, Length:1}}}, Length:3}}

Considerations:

• Sub-messages: Recursive parsing of bytes payloads identified as messages.
• Repeated fields: Handled based on identical field tag numbers.
• Maps: Assumed to be similar to repeated k/v pairs.
• Oneofs: Union type information is lost, only the actual value is parsed.

While this code does not provide a complete production-grade solution, it offers a method to parse unknown proto payloads while preserving some semantic information.

The above is the detailed content of How Can I Parse Unknown Protobuf Messages Without Type Information?. For more information, please follow other related articles on the PHP Chinese website!