When we think about communication between services/microservices, the first option that comes to mind is good old JSON. And this is not without reason, as the format has advantages, such as:
- is easily readable by both computers and humans;
- all modern programming languages can read and generate JSON;
- it is much less verbose than the previous alternative, Jurassic XML.
Using JSON is the recommendation for the vast majority of APIs developed in companies' daily lives. But in some cases, where performance is critical, we may need to consider other alternatives. This post aims to show two alternatives to JSON when it comes to communication between applications.
But what's the problem with JSON? One of its advantages is that it is "easily readable by humans," but this can be a weak point in performance. The fact is that we need to convert the JSON content to some structure that is known by the programming language we are using. An exception to this rule is if we use JavaScript, as JSON is native to it. But if you are using another language, Go, for example, we need to parse the data, as we can see in the (incomplete) code example below:
type event struct { ID uuid.UUID Type string `json:"type"` Source string `json:"source"` Subject string `json:"subject"` Time string `json:"time"` Data string `json:"data"` } var e event err := json.NewDecoder(data).Decode(&e) if err != nil { http.Error(w, err.Error(), http.StatusBadRequest) }
To solve this problem, we can test two alternatives, Protocol Buffers and Flatbuffers.
Protocol Buffers
Protobuf (Protocol Buffers), created by Google, is, according to the official website :
Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data—think XML, but smaller, faster, and more straightforward. You define how you want your data to be structured once. Then, you can use specially generated source code to quickly write and read your structured data to and from various data streams using a variety of languages.
Generally used in conjunction with gRPC (but not necessarily), Protobuf is a binary protocol that significantly increases performance compared to the text format of JSON. But it "suffers" from the same problem as JSON: we need to parse it to a data structure of our language. For example, in Go:
//generated code type Event struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache unknownFields protoimpl.UnknownFields Type string `protobuf:"bytes,1,opt,name=type,proto3" json:"type,omitempty"` Subject string `protobuf:"bytes,2,opt,name=subject,proto3" json:"subject,omitempty"` Source string `protobuf:"bytes,3,opt,name=source,proto3" json:"source,omitempty"` Time string `protobuf:"bytes,4,opt,name=time,proto3" json:"time,omitempty"` Data string `protobuf:"bytes,5,opt,name=data,proto3" json:"data,omitempty"` } e := Event{} err := proto.Unmarshal(data, &e) if err != nil { http.Error(w, err.Error(), http.StatusBadRequest) }
Adopting a binary protocol gives us a performance gain, but we still need to solve the problem of data parsing. Our third competitor focuses on solving this problem.
Flatbuffers
According to the official website:
FlatBuffers is an efficient cross-platform serialization library for C++, C#, C, Go, Java, Kotlin, JavaScript, Lobster, Lua, TypeScript, PHP, Python, Rust, and Swift. It was initially created at Google for game development and other performance-critical applications.
Although initially created for game development, it fits perfectly into the environment we study in this post. Its advantage is that we do not need to parse the data in addition to being a binary protocol. For example, in Go:
//generated code e := events.GetRootAsEvent(data, 0) //we can use the data directly saveEvent(string(e.Type()), string(e.Source()), string(e.Subject()), string(e.Time()), string(e.Data()))
But how much more performant are the two alternatives to JSON? Let's investigate...
Application
The first question that came to my mind was, "how can I apply this in a real scenario?". I imagined the following scenario:
A company with a mobile application, accessed daily by millions of customers, with an internal microservices architecture and which needs to save events generated by users and systems for auditing purposes.
This is a genuine scenario. So real that I live with it every day at the company where I work :)
Note: the scenario above is a simplification and does not represent the actual complexity of the team's application. It serves educational purposes.
The first step is defining an event in Protocol Buffers and Flatbuffers. Both have their own language for defining schemas, which we can then use to generate code in the languages we will use. I will not delve into the details of each scheme as this is easily found in the documentation.
The file event.proto has the Protocol Buffer definition:
syntax = "proto3"; package events; option go_package = "./events_pb"; message Event { string type = 1; string subject = 2; string source = 3; string time = 4; string data = 5; }
And the file event.fbs has the equivalent in Flatbuffers:
namespace events; table Event { type: string; subject:string; source:string; time:string; data:string; } root_type Event;
The next step is to use these definitions to generate the necessary code. The following commands install the dependencies on macOS:
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest brew install protobuf protoc -I=. --go_out=./ event.proto brew install flatbuffers flatc --go event.fbs
The result is the creation of Go packages to manipulate data in each format.
With the requirements met, the next step was implementing the event API. The main.go looked like this:
package main import ( "fmt" "net/http" "os" "github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5/middleware" "github.com/google/uuid" ) func main() { r := handlers() http.ListenAndServe(":3000", r) } func handlers() *chi.Mux { r := chi.NewRouter() if os.Getenv("DEBUG") != "false" { r.Use(middleware.Logger) } r.Post("/json", processJSON()) r.Post("/fb", processFB()) r.Post("/pb", processPB()) return r } func saveEvent(evType, source, subject, time, data string) { if os.Getenv("DEBUG") != "false" { id := uuid.New() q := fmt.Sprintf("insert into event values('%s', '%s', '%s', '%s', '%s', '%s')", id, evType, source, subject, time, data) fmt.Println(q) } // save event to database }
For better organization, I created files to separate each function, which looked as follows:
package main import ( "encoding/json" "net/http" "github.com/google/uuid" ) type event struct { ID uuid.UUID Type string `json:"type"` Source string `json:"source"` Subject string `json:"subject"` Time string `json:"time"` Data string `json:"data"` } func processJSON() http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { var e event err := json.NewDecoder(r.Body).Decode(&e) if err != nil { http.Error(w, err.Error(), http.StatusBadRequest) } saveEvent(e.Type, e.Source, e.Subject, e.Time, e.Data) w.WriteHeader(http.StatusCreated) w.Write([]byte("json received")) } }
package main import ( "io" "net/http" "github.com/eminetto/post-flatbuffers/events_pb" "google.golang.org/protobuf/proto" ) func processPB() http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { body := r.Body data, _ := io.ReadAll(body) e := events_pb.Event{} err := proto.Unmarshal(data, &e) if err != nil { http.Error(w, err.Error(), http.StatusBadRequest) } saveEvent(e.GetType(), e.GetSource(), e.GetSubject(), e.GetTime(), e.GetData()) w.WriteHeader(http.StatusCreated) w.Write([]byte("protobuf received")) } }
package main import ( "io" "net/http" "github.com/eminetto/post-flatbuffers/events" ) func processFB() http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { body := r.Body data, _ := io.ReadAll(body) e := events.GetRootAsEvent(data, 0) saveEvent(string(e.Type()), string(e.Source()), string(e.Subject()), string(e.Time()), string(e.Data())) w.WriteHeader(http.StatusCreated) w.Write([]byte("flatbuffer received")) } }
In the functions processPB() and processFB(), we can see how the generated packages are used to manipulate the data.
Benchmark
The last step of our proof of concept is generating the benchmark to compare the formats. I used the Go stdlib benchmark package for this.
The file main_test.go has tests for each format:
package main import ( "bytes" "fmt" "net/http" "net/http/httptest" "os" "strings" "testing" "github.com/eminetto/post-flatbuffers/events" "github.com/eminetto/post-flatbuffers/events_pb" flatbuffers "github.com/google/flatbuffers/go" "google.golang.org/protobuf/proto" ) func benchSetup() { os.Setenv("DEBUG", "false") } func BenchmarkJSON(b *testing.B) { benchSetup() r := handlers() payload := fmt.Sprintf(`{ "type": "button.clicked", "source": "Login", "subject": "user1000", "time": "2018-04-05T17:31:00Z", "data": "User clicked because X"}`) for i := 0; i <p>It generates an event in each format and sends it to the API.</p> <p>When we run the benchmark, we have the following result:<br> </p> <pre class="brush:php;toolbar:false">Running tool: /opt/homebrew/bin/go test -benchmem -run=^$ -coverprofile=/var/folders/vn/gff4w90d37xbfc_2tn3616h40000gn/T/vscode-gojAS4GO/go-code-cover -bench . github.com/eminetto/post-flatbuffers/cmd/api -failfast -v goos: darwin goarch: arm64 pkg: github.com/eminetto/post-flatbuffers/cmd/api BenchmarkJSON BenchmarkJSON-8 658386 1732 ns/op 2288 B/op 26 allocs/op BenchmarkFlatBuffers BenchmarkFlatBuffers-8 1749194 640.5 ns/op 1856 B/op 21 allocs/op BenchmarkProtobuffer BenchmarkProtobuffer-8 1497356 696.9 ns/op 1952 B/op 21 allocs/op PASS coverage: 77.5% of statements ok github.com/eminetto/post-flatbuffers/cmd/api 5.042s
If this is the first time you have analyzed the results of a Go benchmark, I recommend reading this post, where the author describes the details of each column and its meaning.
To make it easier to visualize, I created graphs for the most critical information generated by the benchmark:
Number of iterations (higher is better)
Nanoseconds per operation (lower is better)
Number of bytes allocated per operation (lower is better)
Number of allocations per operation (lower is better)
Conclusion
The numbers show a great advantage of binary protocols over JSON, especially Flatbuffers. This advantage is that we do not need to parse the data into structures of the language we are using.
Should you refactor your applications to replace JSON with Flatbuffers? Not necessarily. Performance is just one factor that teams must consider when selecting a communication protocol between their services and applications. But if your application receives billions of requests per day, performance improvements like those presented in this post can make a big difference in terms of costs and user experience.
The codes presented here can be found in this repository. I made the examples using the Go language, but both Protocol Buffers and Flatbuffers support different programming languages, so I would love to see other versions of these comparisons. Additionally, other benchmarks can be used, such as network consumption, CPU, etc. (since we only compare memory here).
I hope this post serves as an introduction to these formats and an incentive for new tests and experiments.
Originally published at https://eltonminetto.dev on August 05, 2024
The above is the detailed content of JSON vs FlatBuffers vs Protocol Buffers. For more information, please follow other related articles on the PHP Chinese website!

This article explains Go's package import mechanisms: named imports (e.g., import "fmt") and blank imports (e.g., import _ "fmt"). Named imports make package contents accessible, while blank imports only execute t

This article explains Beego's NewFlash() function for inter-page data transfer in web applications. It focuses on using NewFlash() to display temporary messages (success, error, warning) between controllers, leveraging the session mechanism. Limita

This article details efficient conversion of MySQL query results into Go struct slices. It emphasizes using database/sql's Scan method for optimal performance, avoiding manual parsing. Best practices for struct field mapping using db tags and robus

This article demonstrates creating mocks and stubs in Go for unit testing. It emphasizes using interfaces, provides examples of mock implementations, and discusses best practices like keeping mocks focused and using assertion libraries. The articl

This article explores Go's custom type constraints for generics. It details how interfaces define minimum type requirements for generic functions, improving type safety and code reusability. The article also discusses limitations and best practices

This article details efficient file writing in Go, comparing os.WriteFile (suitable for small files) with os.OpenFile and buffered writes (optimal for large files). It emphasizes robust error handling, using defer, and checking for specific errors.

The article discusses writing unit tests in Go, covering best practices, mocking techniques, and tools for efficient test management.

This article explores using tracing tools to analyze Go application execution flow. It discusses manual and automatic instrumentation techniques, comparing tools like Jaeger, Zipkin, and OpenTelemetry, and highlighting effective data visualization


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

SAP NetWeaver Server Adapter for Eclipse
Integrate Eclipse with SAP NetWeaver application server.

MinGW - Minimalist GNU for Windows
This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

VSCode Windows 64-bit Download
A free and powerful IDE editor launched by Microsoft

MantisBT
Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

mPDF
mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),
