From Zero to Merge: Building a JSON Renaming Field Component in Go

Introduction to Instill-ai

Working on Instill’s pipeline-backend project was like solving a jigsaw ? puzzle—except some pieces kept changing names! My mission? To create a component that could rename JSON fields without creating conflicts. Join me as I share my journey through learning Go, studying Instill’s well-organized docs, and creating a solution that’s now merged and ready to roll! ?

---

The Challenge

Instill needed a way to rename fields in JSON data structures dynamically. The twist? We had to handle cases where a renamed field might clash with an existing field. Without a conflict resolution system, chaos would reign supreme!

instill-ai / pipeline-backend

⇋ A REST/gRPC server for Instill VDP API service

pipeline-backend

pipeline-backend manages all pipeline resources within Versatile Data Pipeline (VDP) to streamline data from the start component, through AI/Data/Application components and to the end component.

Concepts

Pipeline

In ? Instill VDP, a pipeline is a DAG (Directed Acyclic Graph) consisting of multiple components.

flowchart LR
s[Trigger] --> c1[OpenAI Component]
c1 --> c2[Stability AI Component]
c1 --> c3[MySQL Component]
c1 --> e[Response]
c2 --> e

Loading

Component

A Component serves as an essential building block within a Pipeline.

See the component package documentation for more details.

Recipe

A pipeline recipe specifies how components are configured and how they are interconnected.

Recipes are defined in YAML language:

flowchart LR
s[Trigger] --> c1[OpenAI Component]
c1 --> c2[Stability AI Component]
c1 --> c3[MySQL Component]
c1 --> e[Response]
c2 --> e

…

View on GitHub

To be honest, I was starting to doubt if I could solve this issue, but then Anni dropped the perfect message that kept me going.

Once I got comfortable, ChunHao, who had crafted a JSON schema for this task, gave me the green light? to start coding. And so, the journey began!

---

Step 2️⃣: Designing the Solution

The key requirements were:

1. Dynamic Renaming: Fields should be renamed without disturbing the JSON structure.
2. Conflict Detection: We needed to spot any conflicts between the original and renamed fields.
3. Conflict Resolution: A smooth solution, like appending a suffix, would prevent name clashes.

---

Step 3️⃣: Building the Component

Armed with coffee☕ and courage?, I got down to coding. Here’s a sneak peek at the core logic:

Mapping Fields

First, I created a mapping system to track old and new field names. This was key to detecting conflicts.

flowchart LR
s[Trigger] --> c1[OpenAI Component]
c1 --> c2[Stability AI Component]
c1 --> c3[MySQL Component]
c1 --> e[Response]
c2 --> e

Any time a conflict was detected, the function added "_conflict" to the new name. It’s a simple trick that ensures our JSON fields stay unique and, most importantly, friendly to each other! ✌️

Renaming Fields

Once the field mappings were in place, the next step was applying them to our JSON data.

variable
  <span># pipeline input fields</span>
output:
  <span># pipeline output fields</span>
component:
  <component-id>:
    type: <component-definition-id>
    task: <task-id>
    input:
      <span># values for the input fields</span>
    condition: <condition> <span># conditional statement to execute or bypass the</span></condition></task-id></component-definition-id></component-id>

Here’s the logic that applies the mapped names to our JSON data. The result? Our data’s neatly renamed, conflicts resolved, and structure intact. ?

After creating the component dropped the draft PR & got a comment:

---

Step 4️⃣: Testing and Refinement

After familiarizing myself with Instill's testing methods and learning how to create effective test cases, I proceeded further.

Testing time! ? I wrote tests covering everything from simple renames to complex edge cases with nested JSON fields. Each round of testing led to further refinements.

func mapFields(fields map[string]string) map[string]string {
    newFieldMap := make(map[string]string)
    for oldName, newName := range fields {
        // Check for conflict
        if _, exists := newFieldMap[newName]; exists {
            newName += "_conflict" // Add suffix for conflicts
        }
        newFieldMap[oldName] = newName
    }
    return newFieldMap
}

Here’s where I’d love to share a personal reflection: Testing was the hardest part of this project ?‍?. There were times when I thought, "Is this test even doing what it’s supposed to?"

Just then, I ran into a lint issue—

He pointed out the problem and even provided the solution. All I had to do was implement it, but it was a reminder that even the smallest details matter in making the code work smoothly.

Once I got past those initial hurdles, testing became my safety net. It gave me the confidence to know that my code would work across different scenarios ?️‍♂️. It also showed me that testing isn’t just a step to check off—it’s a way to ensure my code is reliable and resilient.

---

Step 5️⃣: CI Check and Final Adjustments

After completing my tests, I pushed my code, ready for the review process. However, our CI (Continuous Integration) checks didn’t pass. Anni’s comment gave me a gentle reminder to double-check my test cases:

“Hey @AkashJana18, could you check your test cases? Our CI check shows it has not passed here. Please test it locally first before pushing it to the PR. Whenever you push your commit, we’ll trigger the checks so you can spot any issues before our engineers review your code. Thanks!”

That’s when I realized I had to run the tests locally before submitting. ChunHao also added:

"Please run and pass it before you request the review. Run $ go test ./pkg/component/operator/json/v0/... to check it locally."

I quickly ran the tests locally, identified the issues, and fixed them.

A little moment of celebration ?

This process made me appreciate the importance of local testing even more, as it ensured everything was solid before submitting for review.

Before merging, ChunHao did a final review, made a few tweaks, QAed Test Recipe and updated the documentation to reflect the new changes. Big thanks to Anni for her ongoing support throughout the process—it made a huge difference. ?

---

? Reflection on the Collaborative Process ??‍??

One of the biggest lessons I learned was how collaboration and mentorship can make or break a project. Instill's moderators, Anni and ChunHao, provided me with the guidance I needed when I was lost in Go syntax or struggling with the right approach. Working together, we turned a complex problem into a clean, functional solution.

I’ll be honest, there were moments I felt like I had bitten off more than I could chew. But the constant encouragement from Anni, combined with the clear direction from ChunHao, kept me on track.

---

⏭️ Next Steps and Future Improvements

Another step could be expanding this approach to other parts of the pipeline that require dynamic field name handling—because who doesn’t love a little bit of automation⚙️?

---

?️ Tools & Resources ?

1. Go Documentation: For diving into Go syntax and understanding core concepts.
2. Instill Docs: A goldmine of well-organized resources to understand the Instill pipeline.
3. Go Testing Framework: The built-in testing package in Go for writing unit tests, ensuring everything works as expected, and integrating with CI tools.
4. Golangci-lint: A Go linters aggregator to identify issues and enforce code quality during development and CI checks.

---

??‍??️ My Learning

With Instill’s rock-solid documentation, guidance from ChunHao, and Anni's moral support, this project became a fantastic learning experience. I went from knowing nothing about Go to implementing a fully functional feature ready for production (and I have the merged PR to prove it ?).

Proof:

The above is the detailed content of From Zero to Merge: Building a JSON Renaming Field Component in Go. For more information, please follow other related articles on the PHP Chinese website!