gRPC Communication Between Go and Python

gRPC is a powerful, high-performance Remote Procedure Call (RPC) framework that, despite being less commonly used than REST, offers significant advantages in certain scenarios.

In addition it's language agnostic and can run in any environment, making it an ideal choice for server-to-server communication.

I will not delve into in whole explenation of it but here is a general link of gRPC. I'll provide a hands on turtorial

Go gRPC client 

Lets image our Go is client but is a server asfor frontend app React, Svelte etc.

func getFirstArg() (string, error) {
    if len(os.Args) < 2 {
        return "", fmt.Errorf("expected 1 argument, but got none")
    }
    return os.Args[1], nil
}

func main() {
    filePath, err := getFirstArg()
    if err != nil {
        log.Fatalf("Failed to get file path from arguments: %v", err)
    }

    fileData, err := ioutil.ReadFile(filePath)
    if err != nil {
        log.Fatalf("Failed to read file: %v", err)
    }

 ...
}

---

As an example React frontend uploads a file, Go process it but we need answers from excel we will use GPT API. While it can be done with Go, Python on the otherhand has more packages that can ease our lives like langchan_openai, pandas for excel and so forth.

---

Lets start with instalation of gRPC preferably in your virtualenv .venv

$ go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
$ export PATH="$PATH:$(go env GOPATH)/bin"

Next up you should install protocol buffer in your OS can follow it here.
Let's create a proto dir where you will store your protocol buffer file I will name it as excel.proto and paste this:

syntax = "proto3";
option go_package = "client-gRPC/proto";
service ExcelService {
    rpc UploadFile(FileRequest) returns (FileResponse);
}
message FileRequest {
    string file_name = 1;
    bytes file_content = 2;
}
message FileResponse {
    bytes file_content = 1;
}

This gRPC service, ExcelService, allows clients to upload a file by sending its name and content. The server responds with the same file content. 

For Go its essential to pass in go_package in Python the line is not needed.

vscode-proto3 is a good extension to download if you use VSCode.

After all of this you can generate your proto files I preffer it in same level as prot dir, for that run this command:

protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative proto/excel.proto

If succesfull two files should be generated, optionally if there would be a lot of adjustments add a Makefile and define it as proto + upper command.

import (
    ....

    "google.golang.org/grpc"
    pb "client-gRPC/proto"
    "github.com/xuri/excelize/v2"
)

func main() {
    ....

    conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())
    if err != nil {
        log.Fatalf("Failed to connect to gRPC server: %v", err)
    }
    defer conn.Close()

    client := pb.NewExcelServiceClient(conn)

    req := &pb.FileRequest{
        FileName:    filePath,
        FileContent: fileData,
    }

    res, err := client.UploadFile(context.Background(), req)
    if err != nil {
        log.Fatalf("Failed to upload file: %v", err)
    }

    outputFile := "output.xlsx"
    err = saveBytesAsExcel(outputFile, res.FileContent)
    if err != nil {
        log.Fatalf("Failed to save bytes as Excel file: %v", err)
    }

    fmt.Printf("Excel file saved as: %s\n", outputFile)
}

func saveBytesAsExcel(filePath string, fileContent []byte) error {
    f, err := excelize.OpenReader(bytes.NewReader(fileContent))
    if err != nil {
        return fmt.Errorf("failed to open Excel file: %v", err)
    }

    if err := f.SaveAs(filePath); err != nil {
        return fmt.Errorf("failed to save Excel file: %v", err)
    }
    return nil
}

We make a connection to listen to 50051 that will be our Python server, &pb.FileRequest was generated prior by using proto command and now we are importing the methods. If you run you will recive ? due to Python server not established yet.

Failed to upload file: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:50051: connect: connection refused"

Python gRPC server

As python will act as a server the approach will be slightly different but in essense same proto file appart from package field is not reqired. Lets start by creating a base main.py without the gRPC just to give a glance of how GPT will populate the questions in excel.

import os
import openai
import pandas as pd
from dotenv import load_dotenv

def get_answer_from_gpt(apikey: str, question: str):
    openai.api_key = apikey
    response = openai.ChatCompletion.create(
        model="gpt-4",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": question}
        ]
    )
    return response['choices'][0]['message']['content'].strip()

def answer_questions_df(df: pd.DataFrame, apikey: str):
    answers = []

    for question in df.iloc[:, 0]: 
        answer = get_answer_from_gpt(apikey, question)
        answers.append(answer)
    return answers

if __name__ == "__main__":
    load_dotenv()

    openai_api_key = os.getenv("OPENAI_API_KEY", "OpenAI API key hasn't been set.")

    df = pd.read_excel('Book1.xlsx')

    df['Answer'] = answer_questions_df(df, openai_api_key

---

Its a simple script that will answer questions that Go will send us but the LOC is less due to dedicated openai library that makes it easier.

---

We start by as well adding proto dir with same file as above the option section can be removed as disccused. Install gRPC in your virtualenv preferably and follow here the instalation for proto generation I ran"

python3 -m grpc_tools.protoc --proto_path=proto --python_out=proto --grpc_python_out=proto proto/excel.proto

To be in same lvl as my proto directory remember to add __init.py!

Ones the files have been generated lets continue on.

import io
import grpc
from proto import excel_pb2_grpc as excel_grpc
from proto import excel_pb2

class ExcelService(excel_grpc.ExcelServiceServicer):
    def UploadFile(self, request, context):
        try:
            # Convert bytes to a file-like object
            file_like_object = io.BytesIO(request.file_content)

            # Load the workbook from the file-like object
            workbook = openpyxl.load_workbook(file_like_object)

            # Access the first sheet (or use appropriate logic to get the sheet you need)
            sheet = workbook.active

            # Convert the sheet to a DataFrame
            data = sheet.values
            columns = next(data)  # Get the header row
            df = pd.DataFrame(data, columns=columns)

            print("Loaded DataFrame:")
            print(df.head())

            # Ensure that the DataFrame is not empty and has questions
            if df.empty or df.shape[1] < 1:
                print("DataFrame is empty or does not have the expected columns.")
                return excel_pb2.FileResponse(file_content=b'')

            # Get answers and add them to the DataFrame
            answers = answer_questions_df(df, openai_api_key)
            df['Answer'] = answers

            # Write the updated DataFrame back to a BytesIO object
            output = io.BytesIO()
            with pd.ExcelWriter(output, engine='openpyxl') as writer:
                df.to_excel(writer, index=False, sheet_name='Sheet1')

            # Reset the buffer's position to the beginning
            output.seek(0)

            # Return the modified file content
            response = excel_pb2.FileResponse(file_content=output.read())
            return response
        except Exception as e:
            print(f"Error processing file: {e}")
            return excel_pb2.FileResponse(file_content=b'')

def serve():
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    excel_grpc.add_ExcelServiceServicer_to_server(ExcelService(), server)
    server.add_insecure_port('[::]:50051')
    server.start()
    print("Server running on port 50051.")
    server.wait_for_termination()

if __name__ == "__main__":
    load_dotenv()

    openai_api_key = os.getenv("OPENAI_API_KEY", "OpenAI API key hasn't been set.")

    serve()

We define server and add ExcelService class what containes the methods generated by proto file. Because we recive file by bytes have to use io byte reader and commence further processing of the file and population the second column.

response = excel_pb2.FileResponse(file_content=output.read())

At the end we are returning ☝️ for our Go client to recive.

To be able to find proto files in python however you should define an export path

export PYTHONPATH=$PYTHONPATH:mnt/c/own_dev/gRPC/server/proto

Running Client and Server

If all is good you can run

#First comes server

python3 -m main

#Then client

go run client.go Book1.xlsx

And you should get the updated .xlsx file in Go client side.

Conclusion

In this article we explored the fundamentals of setting up gRPC communication between Python server and Go client. By leveraging gRPC, we established a seamless way to send an Excel file from a Go application to a Python server, process the file using OpenAI's GPT API, and return the modified file back to the Go client.

The above is the detailed content of gRPC Communication Between Go and Python. For more information, please follow other related articles on the PHP Chinese website!