How Can I Parse Unknown Protobuf Messages Without Type Information?

Unknown Proto Message Demarshalling Without Type Information

While protobuf's proto.Unmarshal() method requires a known message type, unknown messages present a challenge. However, we can extract limited information using the protowire package.

Approach:

1. Use protowire.ConsumeField() to iterate through the unknown message payload.

2. Identify the field type, which could be:

   • Varint
   • Fixed64
   • Bytes
   • StartGroup
   • Fixed32
3. Parse the field payload based on its type.
4. Handle sub-messages by recursively parsing their payloads.

Data Model:

type Field struct {
    Tag Tag
    Val Val
}

type Tag struct {
    Num int32
    Type protowire.Type
}

type Val struct {
    Payload interface{}
    Length int
}

Parser:

func parseUnknown(b []byte) []Field {
    // Iteratively consume and parse fields
    for len(b) > 0 {
        // Read field tag and length
        n, t, fieldlen := protowire.ConsumeField(b)
        if fieldlen < 1 {
            return nil
        }
        field := Field{Tag: Tag{Num: int32(n), Type: t}}

        // Read and process tag and value content
        _, _, taglen := protowire.ConsumeTag(b[:fieldlen])
        if taglen < 1 {
            return nil
        }
        var v interface{}
        var vlen int
        switch t {
            case protowire.VarintType:
                v, vlen = protowire.ConsumeVarint(b[taglen:fieldlen])
            case protowire.Fixed64Type:
                v, vlen = protowire.ConsumeFixed64(b[taglen:fieldlen])
            case protowire.BytesType:
                v, vlen = protowire.ConsumeBytes(b[taglen:fieldlen])
                sub := parseUnknown(v.([]byte))
                if sub != nil {
                    v = sub
                }
            case protowire.StartGroupType:
                v, vlen = protowire.ConsumeGroup(n, b[taglen:fieldlen])
                sub := parseUnknown(v.([]byte))
                if sub != nil {
                    v = sub
                }
            case protowire.Fixed32Type:
                v, vlen = protowire.ConsumeFixed32(b[taglen:fieldlen])
        }
        if vlen < 1 {
            return nil
        }

        field.Val = Val{Payload: v, Length: vlen - taglen}
        fields = append(fields, field)
        b = b[fieldlen:]
    }
    return fields
}

Example (Message Structure):

message Foo {
  string a = 1;
  string b = 2;
  Bar bar = 3;
}

message Bar {
  string c = 1;
}

Example (Parsed Output):

main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x41}, Length:1}}
main.Field{Tag:main.Tag{Num:2, Type:2}, Val:main.Val{Payload:[]uint8{0x42}, Length:1}}
main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x43}, Length:1}}
main.Field{Tag:main.Tag{Num:3, Type:2}, Val:main.Val{Payload:[]main.Field{main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x43}, Length:1}}}, Length:3}}

Considerations:

• Sub-messages: Recursive parsing of bytes payloads identified as messages.
• Repeated fields: Handled based on identical field tag numbers.
• Maps: Assumed to be similar to repeated k/v pairs.
• Oneofs: Union type information is lost, only the actual value is parsed.

While this code does not provide a complete production-grade solution, it offers a method to parse unknown proto payloads while preserving some semantic information.

The above is the detailed content of How Can I Parse Unknown Protobuf Messages Without Type Information?. For more information, please follow other related articles on the PHP Chinese website!