search
HomeBackend DevelopmentGolangGoroutine leaks when trying `ReadAll(response.Body)` where `response` is returned by http `client.Do(...)`

当尝试 `ReadAll(response.Body)` 时,goroutine 泄漏,其中 `response` 由 http `client.Do(...)` 返回

#php editor Baicao discovered a common goroutine leak when solving programming problems. When we try to use `ReadAll(response.Body)`, the returned `response` object is returned by the `client.Do(...)` method of the `http` package. However, this operation can lead to goroutine leakage problems. In order to avoid this happening, we need to take some measures to properly handle http responses.

Question content

The question comes from a very common scenario where all bytes are read from the response body after an http method call.

With the help of https://github.com/uber-go/goleak, I discovered an interesting goroutine leak.

To demonstrate the issue, run the following test code using go test, which results in found an unexpected goroutine:

package main

import (
    "io"
    "net/http"
    "testing"

    "go.uber.org/goleak"
    "gotest.tools/v3/assert"
)

func testleak(t *testing.t) {
    defer goleak.verifynone(t)

    request, err := http.newrequest(http.methodget, "https://google.com", nil)
    assert.nilerror(t, err)

    client := http.client{}

    response, err := client.do(request)
    assert.nilerror(t, err)

    defer response.body.close()

    _, err = io.readall(response.body)

    assert.nilerror(t, err)
}

Full test output:

$ go test
--- FAIL: TestLeak (1.26s)
    main_test.go:28: found unexpected goroutines:
        [Goroutine 14 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
        goroutine 14 [IO wait]:
        internal/poll.runtime_pollWait(0x7f945c1e0398, 0x72)
                /home/wxh/.local/go/src/runtime/netpoll.go:305 +0x89
        internal/poll.(*pollDesc).wait(0xc000154280?, 0xc0000ca000?, 0x0)
                /home/wxh/.local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
        internal/poll.(*pollDesc).waitRead(...)
                /home/wxh/.local/go/src/internal/poll/fd_poll_runtime.go:89
        internal/poll.(*FD).Read(0xc000154280, {0xc0000ca000, 0x2000, 0x2000})
                /home/wxh/.local/go/src/internal/poll/fd_unix.go:167 +0x25a
        net.(*netFD).Read(0xc000154280, {0xc0000ca000?, 0xc00017edd8?, 0xc0004d5808?})
                /home/wxh/.local/go/src/net/fd_posix.go:55 +0x29
        net.(*conn).Read(0xc00009c000, {0xc0000ca000?, 0xc0000c68f0?, 0x2c?})
                /home/wxh/.local/go/src/net/net.go:183 +0x45
        crypto/tls.(*atLeastReader).Read(0xc0002d4138, {0xc0000ca000?, 0x0?, 0x29f6540f238f2947?})
                /home/wxh/.local/go/src/crypto/tls/conn.go:787 +0x3d
        bytes.(*Buffer).ReadFrom(0xc0000a0278, {0x7c3860, 0xc0002d4138})
                /home/wxh/.local/go/src/bytes/buffer.go:202 +0x98
        crypto/tls.(*Conn).readFromUntil(0xc0000a0000, {0x7c3b80?, 0xc00009c000}, 0xc0000a41c0?)
                /home/wxh/.local/go/src/crypto/tls/conn.go:809 +0xe5
        crypto/tls.(*Conn).readRecordOrCCS(0xc0000a0000, 0x0)
                /home/wxh/.local/go/src/crypto/tls/conn.go:616 +0x116
        crypto/tls.(*Conn).readRecord(...)
                /home/wxh/.local/go/src/crypto/tls/conn.go:582
        crypto/tls.(*Conn).Read(0xc0000a0000, {0xc0001cf000, 0x1000, 0x11?})
                /home/wxh/.local/go/src/crypto/tls/conn.go:1287 +0x16f
        bufio.(*Reader).Read(0xc0001af1a0, {0xc0001aad60, 0x9, 0xc0004d5d18?})
                /home/wxh/.local/go/src/bufio/bufio.go:237 +0x1bb
        io.ReadAtLeast({0x7c3760, 0xc0001af1a0}, {0xc0001aad60, 0x9, 0x9}, 0x9)
                /home/wxh/.local/go/src/io/io.go:332 +0x9a
        io.ReadFull(...)
                /home/wxh/.local/go/src/io/io.go:351
        net/http.http2readFrameHeader({0xc0001aad60?, 0x9?, 0xc0004d5df0?}, {0x7c3760?, 0xc0001af1a0?})
                /home/wxh/.local/go/src/net/http/h2_bundle.go:1565 +0x6e
        net/http.(*http2Framer).ReadFrame(0xc0001aad20)
                /home/wxh/.local/go/src/net/http/h2_bundle.go:1829 +0x95
        net/http.(*http2clientConnReadLoop).run(0xc0004d5f98)
                /home/wxh/.local/go/src/net/http/h2_bundle.go:8875 +0x130
        net/http.(*http2ClientConn).readLoop(0xc00009e180)
                /home/wxh/.local/go/src/net/http/h2_bundle.go:8771 +0x6f
        created by net/http.(*http2Transport).newClientConn
                /home/wxh/.local/go/src/net/http/h2_bundle.go:7478 +0xaaa
        
         Goroutine 37 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
        goroutine 37 [IO wait]:
        internal/poll.runtime_pollWait(0x7f945c1e02a8, 0x72)
                /home/wxh/.local/go/src/runtime/netpoll.go:305 +0x89
        internal/poll.(*pollDesc).wait(0xc0003ca000?, 0xc000202000?, 0x0)
                /home/wxh/.local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
        internal/poll.(*pollDesc).waitRead(...)
                /home/wxh/.local/go/src/internal/poll/fd_poll_runtime.go:89
        internal/poll.(*FD).Read(0xc0003ca000, {0xc000202000, 0x1300, 0x1300})
                /home/wxh/.local/go/src/internal/poll/fd_unix.go:167 +0x25a
        net.(*netFD).Read(0xc0003ca000, {0xc000202000?, 0xc0004e6798?, 0xc0004c0808?})
                /home/wxh/.local/go/src/net/fd_posix.go:55 +0x29
        net.(*conn).Read(0xc000014028, {0xc000202000?, 0xc000136630?, 0x2c?})
                /home/wxh/.local/go/src/net/net.go:183 +0x45
        crypto/tls.(*atLeastReader).Read(0xc000276288, {0xc000202000?, 0x0?, 0x150fbb8ccb566149?})
                /home/wxh/.local/go/src/crypto/tls/conn.go:787 +0x3d
        bytes.(*Buffer).ReadFrom(0xc0000a0978, {0x7c3860, 0xc000276288})
                /home/wxh/.local/go/src/bytes/buffer.go:202 +0x98
        crypto/tls.(*Conn).readFromUntil(0xc0000a0700, {0x7c3b80?, 0xc000014028}, 0xc0004cc2c0?)
                /home/wxh/.local/go/src/crypto/tls/conn.go:809 +0xe5
        crypto/tls.(*Conn).readRecordOrCCS(0xc0000a0700, 0x0)
                /home/wxh/.local/go/src/crypto/tls/conn.go:616 +0x116
        crypto/tls.(*Conn).readRecord(...)
                /home/wxh/.local/go/src/crypto/tls/conn.go:582
        crypto/tls.(*Conn).Read(0xc0000a0700, {0xc00023c000, 0x1000, 0x11?})
                /home/wxh/.local/go/src/crypto/tls/conn.go:1287 +0x16f
        bufio.(*Reader).Read(0xc00022a2a0, {0xc00020e2e0, 0x9, 0xc0004c0d18?})
                /home/wxh/.local/go/src/bufio/bufio.go:237 +0x1bb
        io.ReadAtLeast({0x7c3760, 0xc00022a2a0}, {0xc00020e2e0, 0x9, 0x9}, 0x9)
                /home/wxh/.local/go/src/io/io.go:332 +0x9a
        io.ReadFull(...)
                /home/wxh/.local/go/src/io/io.go:351
        net/http.http2readFrameHeader({0xc00020e2e0?, 0x9?, 0xc0004c0df0?}, {0x7c3760?, 0xc00022a2a0?})
                /home/wxh/.local/go/src/net/http/h2_bundle.go:1565 +0x6e
        net/http.(*http2Framer).ReadFrame(0xc00020e2a0)
                /home/wxh/.local/go/src/net/http/h2_bundle.go:1829 +0x95
        net/http.(*http2clientConnReadLoop).run(0xc0004c0f98)
                /home/wxh/.local/go/src/net/http/h2_bundle.go:8875 +0x130
        net/http.(*http2ClientConn).readLoop(0xc00019e300)
                /home/wxh/.local/go/src/net/http/h2_bundle.go:8771 +0x6f
        created by net/http.(*http2Transport).newClientConn
                /home/wxh/.local/go/src/net/http/h2_bundle.go:7478 +0xaaa
        ]
FAIL
exit status 1

Does anyone know how to solve this problem?

Thanks!

Solution

In Go, http.Client uses Transport (http.RoundTripper interface). http.Transport is its default implementation. The default http.Transfer maintains a connection pool that holds connections that remain active or idle for some reason.

Basically all you need to do is make sure there are no connections in the pool* when you call goleak.Verify

This can be done by:

  1. Disable keep-alive for all http clients before performing any requests:
    client := http.Client{Transport: &http.Transport{DisableKeepAlive: true}}
  2. Close idle connections before calling goleak.Verify*:
    client.CloseIdleConnections() // Available since Go 1.12

The above is the detailed content of Goroutine leaks when trying `ReadAll(response.Body)` where `response` is returned by http `client.Do(...)`. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:stackoverflow. If there is any infringement, please contact admin@php.cn delete
How do I write mock objects and stubs for testing in Go?How do I write mock objects and stubs for testing in Go?Mar 10, 2025 pm 05:38 PM

This article demonstrates creating mocks and stubs in Go for unit testing. It emphasizes using interfaces, provides examples of mock implementations, and discusses best practices like keeping mocks focused and using assertion libraries. The articl

How can I define custom type constraints for generics in Go?How can I define custom type constraints for generics in Go?Mar 10, 2025 pm 03:20 PM

This article explores Go's custom type constraints for generics. It details how interfaces define minimum type requirements for generic functions, improving type safety and code reusability. The article also discusses limitations and best practices

How do you write unit tests in Go?How do you write unit tests in Go?Mar 21, 2025 pm 06:34 PM

The article discusses writing unit tests in Go, covering best practices, mocking techniques, and tools for efficient test management.

How can I use tracing tools to understand the execution flow of my Go applications?How can I use tracing tools to understand the execution flow of my Go applications?Mar 10, 2025 pm 05:36 PM

This article explores using tracing tools to analyze Go application execution flow. It discusses manual and automatic instrumentation techniques, comparing tools like Jaeger, Zipkin, and OpenTelemetry, and highlighting effective data visualization

How do you use the pprof tool to analyze Go performance?How do you use the pprof tool to analyze Go performance?Mar 21, 2025 pm 06:37 PM

The article explains how to use the pprof tool for analyzing Go performance, including enabling profiling, collecting data, and identifying common bottlenecks like CPU and memory issues.Character count: 159

Explain the purpose of Go's reflect package. When would you use reflection? What are the performance implications?Explain the purpose of Go's reflect package. When would you use reflection? What are the performance implications?Mar 25, 2025 am 11:17 AM

The article discusses Go's reflect package, used for runtime manipulation of code, beneficial for serialization, generic programming, and more. It warns of performance costs like slower execution and higher memory use, advising judicious use and best

How do you use table-driven tests in Go?How do you use table-driven tests in Go?Mar 21, 2025 pm 06:35 PM

The article discusses using table-driven tests in Go, a method that uses a table of test cases to test functions with multiple inputs and outcomes. It highlights benefits like improved readability, reduced duplication, scalability, consistency, and a

How do you use sync.WaitGroup to wait for multiple goroutines to complete?How do you use sync.WaitGroup to wait for multiple goroutines to complete?Mar 19, 2025 pm 02:51 PM

The article explains how to use sync.WaitGroup in Go to manage concurrent operations, detailing initialization, usage, common pitfalls, and best practices.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
1 months agoBy尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
1 months agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Atom editor mac version download

Atom editor mac version download

The most popular open source editor

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

Safe Exam Browser

Safe Exam Browser

Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

MantisBT

MantisBT

Mantis is an easy-to-deploy web-based defect tracking tool designed to aid in product defect tracking. It requires PHP, MySQL and a web server. Check out our demo and hosting services.

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment