Pipeline Concurrency Pattern in Go: A Comprehensive Visual Guide

⚠️ How to go about this series?

1. Run Every Example: Don't just read the code. Type it out, run it, and observe the behavior.
2. Experiment and Break Things: Remove sleeps and see what happens, change channel buffer sizes, modify goroutine counts.
Breaking things teaches you how they work
3. Reason About Behavior: Before running modified code, try predicting the outcome. When you see unexpected behavior, pause and think why. Challenge the explanations.
4. Build Mental Models: Each visualization represents a concept. Try drawing your own diagrams for modified code.

In our previous post, we explored the Generator concurrency pattern, the building blocks of Go's other concurrency patterns. You can give it a read here:

Generator Concurrency Pattern in Go: A Visual Guide

Souvik Kar Mahapatra ・ Dec 25

#go #tutorial #programming #learning

Now, let's look at how these primitives combine to form powerful patterns that solve real-world problems.

In this post we'll cover Pipeline Pattern and will try to visualize them. So let's gear up as we'll be hands on through out the process.

Pipeline Pattern

A pipeline is like an assembly line in a factory, where each stage performs a specific task on the data and passes the result to the next stage.

We build pipelines by connecting goroutines with channels, where each goroutine represents a stage that receives data, processes it, and sends it to the next stage.

Let's implement a simple pipeline that:

1. Generates numbers
2. Squares them
3. Prints the results

// Stage 1: Generate numbers
func generate(nums ...int) 


<blockquote>
<p>✏️ Quick byte </p>

<p><u><strong></strong></u><br>
A channel of type 

</p>
<p><u><strong>chan int This denotes a bidirectional channel.</strong></u><br>
A channel of type chan int can be used to both send and receive values.</p>
</blockquote>

<p>Let's go ahead and visualize the above example:</p>

<p><img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/173562621693417.jpg?x-oss-process=image/resize,p_40" class="lazy" alt="Pipeline Concurrency Pattern in Go: A Comprehensive Visual Guide"></p>

<p>Here you can see each the building blocks of the pipeline are goroutines following generator pattern. Implies that as soon as the data is ready at any step the next step in the pipeline can start processing it unlike sequential processing. </p>
<h3>
  
  
  Error Handling in Pipelines
</h3>

<p>Core principles should be:</p>

<ol>
<li>Each stage knows exactly what to do with both good and bad values</li>
<li>Errors can't get lost in the pipeline</li>
<li>Bad values don't cause panics</li>
<li>The error message carries context about what went wrong</li>
<li>The pipeline can be extended with more stages, and they'll all handle errors consistently</li>
</ol>

<p>let's update our code with some proper error handling.<br>
</p>
<pre class="brush:php;toolbar:false">type Result struct {
    Value int
    Err   error
}

func generateWithError(nums ...int) 

<h3>
  
  
  Why Use Pipeline Pattern?
</h3>

<p>Let's take an example to understand better, we have a data processing workflow that follows the pipeline pattern as shown below.</p>

<p><img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/173562621771746.jpg?x-oss-process=image/resize,p_40" class="lazy" alt="Pipeline Concurrency Pattern in Go: A Comprehensive Visual Guide"></p><ol>
<li>Each stage in a pipeline operates independently, communicating only through channels. This enables several benefit:</li>
</ol>

<p>? Each stage can be developed, tested, and modified independently<br>
? Changes to one stage's internals don't affect other stages<br>
? Easy to add new stages or modify existing ones<br>
? Clear separation of concerns</p>

<p><img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/173562621851201.jpg?x-oss-process=image/resize,p_40" class="lazy" alt="Pipeline Concurrency Pattern in Go: A Comprehensive Visual Guide"></p>

<ol>
<li>Pipeline patterns naturally enable parallel/concurrent processing. Each stage can process different data simultaneously as soon as the data is available.</li>
</ol>

<p><img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/173562621927218.jpg?x-oss-process=image/resize,p_40" class="lazy" alt="Pipeline Concurrency Pattern in Go: A Comprehensive Visual Guide"></p>

<p>And the best part? We can run multiple instance of each stage (workers) for more concurrent requirements like so:</p>

<p><img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/173562622047220.jpg?x-oss-process=image/resize,p_40" class="lazy" alt="Pipeline Concurrency Pattern in Go: A Comprehensive Visual Guide"></p>

<blockquote>
<p>?? Hey but isn't that the <strong>Fan-In and Fan-Out Concurrency Pattern</strong>?</p>
</blockquote>

<p>Bingo! Good catch right there. It is indeed a Fan-Out, Fan-In pattern, which is a specific type of pipeline pattern. We are going to cover it in details in out next post so fret not ;)</p>
<h3>
  
  
  Real world use case
</h3>

<p><strong>processing images in a pipeline</strong><br>
</p>
<pre class="brush:php;toolbar:false">// Stage 1: Generate numbers
func generate(nums ...int) 


<p>or something as complicated as log processing pipeline</p>

<p><img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/173562622173719.jpg?x-oss-process=image/resize,p_40" class="lazy" alt="Pipeline Concurrency Pattern in Go: A Comprehensive Visual Guide"></p>
<h3>
  
  
  Pipeline scaling patterns
</h3>

<p><img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/173562622294523.jpg?x-oss-process=image/resize,p_40" class="lazy" alt="Pipeline Concurrency Pattern in Go: A Comprehensive Visual Guide"></p>
<h4>
  
  
  <strong>Horizontal Scaling (Fan-Out, Fan-In)</strong>
</h4>

<p>This pattern is ideal for CPU-bound operations where work can be processed independently. The pipeline distributes work across multiple workers and then recombines the results. This is particularly effective when:</p>

<ol>
<li>Processing is CPU-intensive (data transformations, calculations)</li>
<li>Tasks can be processed independently</li>
<li>You have multiple CPU cores available</li>
</ol>
<h4>
  
  
  <strong>Buffered Pipeline</strong>
</h4>

<p>This pattern helps manage speed mismatches between pipeline stages. The buffer acts as a shock absorber, allowing fast stages to work ahead without being blocked by slower stages. This is useful when:</p><ol>
<li>Different stages have varying processing speeds</li>
<li>You want to maintain steady throughput</li>
<li>Memory usage for buffering is acceptable</li>
<li>You need to handle burst processing</li>
</ol>
<h4>
  
  
  <strong>Batched Processing</strong>
</h4>

<p>This pattern optimizes I/O-bound operations by grouping multiple items into a single batch. Instead of processing items one at a time, it collects them into groups and processes them together. This is effective when:</p>

<ol>
<li>You're working with external systems (databases, APIs)</li>
<li>Network round-trips are expensive</li>
<li>The operation has significant fixed overhead per request</li>
<li>You need to optimize throughput over latency</li>
</ol>

<blockquote>
<p>Each of these patterns can be combined as needed. For example, you might use batched processing with horizontal scaling, where multiple workers each process batches of items. <strong>The key is understanding your bottlenecks and choosing the appropriate pattern to address them</strong>.</p>
</blockquote>

<hr>

<p>That wraps up our deep dive into the Generator pattern! Coming up next, we'll explore the <strong>Pipeline concurrency pattern</strong>, where we'll see how to chain our generators together to build powerful data processing flows.</p>

<p>If you found this post helpful, have any questions, or want to share your own experiences with generators - I'd love to hear from you in the comments below. Your insights and questions help make these explanations even better for everyone.</p>

<p>If you missed out visual guide to Golang's goroutine and channels check it out here:</p>


<div>
  
    <div>
      <img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/173562620177807.jpg?x-oss-process=image/resize,p_40" class="lazy" alt="Pipeline Concurrency Pattern in Go: A Comprehensive Visual Guide">
    </div>
  
  
    <div>
      <h2 id="Understanding-and-visualizing-Goroutines-and-Channels-in-Golang">Understanding and visualizing Goroutines and Channels in Golang</h2>
      <h3 id="Souvik-Kar-Mahapatra-Dec">Souvik Kar Mahapatra ・ Dec 20</h3>
      <div>
        #go
        #programming
        #learning
        #tutorial
      </div>
    </div>
  
</div>



<p>Stay tuned for more Go concurrency patterns! ?</p>

<p><img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/173562622776254.jpg?x-oss-process=image/resize,p_40" class="lazy" alt="Pipeline Concurrency Pattern in Go: A Comprehensive Visual Guide"></p>


          

            
  

            
        

The above is the detailed content of Pipeline Concurrency Pattern in Go: A Comprehensive Visual Guide. For more information, please follow other related articles on the PHP Chinese website!