How to Implement Delimited I/O in C for Protocol Buffers?

Delimited I/O Functions in Protocol Buffers: C Equivalents

In scenarios where multiple Protocol Buffers messages are read or written from files in both C and Java, it becomes necessary to attach length prefixes to the messages. While the Java API provides dedicated "Delimited" I/O functions for this purpose, their equivalent counterparts in C may not be readily apparent.

The recent update indicates that such C equivalents now reside in google/protobuf/util/delimited_message_util.h as part of version 3.3.0. However, prior to this update, there were alternative implementations that addressed this requirement efficiently.

One such implementation, provided by a former author of the C and Java protobuf libraries, includes optimizations that prevent potential failures after 64MB of input. These implementations, as depicted below, enforce the 64MB limit on individual messages, ensuring that streaming continues seamlessly without exceeding the overall limit.

Optimized Delimited I/O Implementations for C

<code class="cpp">bool writeDelimitedTo(
    const google::protobuf::MessageLite& message,
    google::protobuf::io::ZeroCopyOutputStream* rawOutput) {
  // Initialize a CodedOutputStream for each message.
  google::protobuf::io::CodedOutputStream output(rawOutput);

  // Determine the message size and write it as a Varint.
  int size = message.ByteSize();
  output.WriteVarint32(size);

  // Optimize for messages fitting into a single buffer.
  uint8_t* buffer = output.GetDirectBufferForNBytesAndAdvance(size);
  if (buffer != NULL) message.SerializeWithCachedSizesToArray(buffer);
  else message.SerializeWithCachedSizes(&output);

  return true;
}

bool readDelimitedFrom(
    google::protobuf::io::ZeroCopyInputStream* rawInput,
    google::protobuf::MessageLite* message) {
  // Initialize a CodedInputStream for each message.
  google::protobuf::io::CodedInputStream input(rawInput);

  // Read the message size.
  uint32_t size;
  if (!input.ReadVarint32(&size)) return false;

  // Restrict reading to the determined message size.
  google::protobuf::io::CodedInputStream::Limit limit =
      input.PushLimit(size);

  // Parse the message and verify it fits within the limit.
  if (!message->MergeFromCodedStream(&input)) return false;
  if (!input.ConsumedEntireMessage()) return false;

  // Lift the reading restriction.
  input.PopLimit(limit);

  return true;
}</code>

These optimized implementations effectively handle messages of varying sizes and provide a reliable solution for delimited I/O in C .

The above is the detailed content of How to Implement Delimited I/O in C for Protocol Buffers?. For more information, please follow other related articles on the PHP Chinese website!