search
HomeBackend DevelopmentGolanghtml rendering function memory leak

html 渲染函数内存泄漏

Question content

The problem I'm facing is that even trying just 200 requests causes the program to occupy 6gb of the container's memory and ends up being oom kill. My idea is to extract all text nodes present in the html and then process them to extract their name, the html and the text of that tag. So, to generate html for a specific tag, I use the render function from golang.org/x/net/html. Where I provide strings.builder as io.writer to write the generated html. But for some reason the builder takes too much memory.

package main

import (
    "encoding/csv"
    "io"
    "log"
    "net/http"
    "strings"
    "golang.org/x/net/html"
)

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/data", GetData)
    if err := http.ListenAndServe(":8001", mux); err != nil {
        log.Println(err)
    }
}

type TagInfo struct {
    Tag  string
    Name string
    Text string
}

// http.handler
func GetData(w http.ResponseWriter, r *http.Request) {
    u := r.URL.Query().Get("url")
    doc, err := GetDoc(u)
    if err != nil {
        log.Println(err)
        w.WriteHeader(500)
        return
    }
    var buf strings.Builder
    data := Extract(doc, &buf)
    csvw := csv.NewWriter(io.Discard)
    for _, d := range data {
        csvw.Write([]string{d.Name, d.Tag, d.Text})
    }
}

// fires request and get text/html
func GetDoc(u string) (*html.Node, error) {
    res, err := http.Get(u)
    if err != nil {
        return nil, err
    }
    defer res.Body.Close()
    return html.Parse(res.Body)
}

func Extract(doc *html.Node, buf *strings.Builder) []TagInfo {
    var (
        tags = make([]TagInfo, 0, 100)
        f    func(*html.Node)
    )

    f = func(n *html.Node) {
        if n.Type == html.TextNode {
            text := strings.TrimSpace(n.Data)
            if text != "" {
                parent := n.Parent
                tag := Render(parent, buf)
                tagInfo := TagInfo{
                    Tag:  tag,
                    Name: parent.Data,
                    Text: n.Data,
                }
                tags = append(tags, tagInfo)
            }
        }
        for child := n.FirstChild; child != nil; child = child.NextSibling {
            f(child)
        }
    }
    f(doc)
    return tags
}

// Render the html around the tag
// if node is text then pass the
// parent node paramter in function
func Render(n *html.Node, buf *strings.Builder) string {
    defer buf.Reset()
    if err := html.Render(buf, n); err != nil {
        log.Println(err)
        return ""
    }
    return buf.String()
}

If you want a specific list of URLs, here it is. I made about 60 requests at one time.

I tried using bytes.buffer bytes.buffer and sync.pool but both have the same problem. Using pprof I noticed that the writestring method of strings.builder caused a lot of memory usage.


Correct answer


So the basic question here is to accept any content-type, which is not acceptable in terms of crawling, most Websites need to send text/html.

The problem is that even if the url sends anything that does not represent html data golang.org/x/net/html it is still accepted without throwing an error.

Let's take an example where application/pdf is returned, then the body will contain html.Parse the binary data of the parsed pdf and no error will be returned, this is Weird-behavior library for scraping/crawling accepted binary data.

The solution is: Check the response headers, if only the data is html, then continue, otherwise there will be ambiguity or higher memory usage (maybe lower), but we can't predict what will happen occur.

The above is the detailed content of html rendering function memory leak. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:stackoverflow. If there is any infringement, please contact admin@php.cn delete
How do you use the pprof tool to analyze Go performance?How do you use the pprof tool to analyze Go performance?Mar 21, 2025 pm 06:37 PM

The article explains how to use the pprof tool for analyzing Go performance, including enabling profiling, collecting data, and identifying common bottlenecks like CPU and memory issues.Character count: 159

What are the vulnerabilities of Debian OpenSSLWhat are the vulnerabilities of Debian OpenSSLApr 02, 2025 am 07:30 AM

OpenSSL, as an open source library widely used in secure communications, provides encryption algorithms, keys and certificate management functions. However, there are some known security vulnerabilities in its historical version, some of which are extremely harmful. This article will focus on common vulnerabilities and response measures for OpenSSL in Debian systems. DebianOpenSSL known vulnerabilities: OpenSSL has experienced several serious vulnerabilities, such as: Heart Bleeding Vulnerability (CVE-2014-0160): This vulnerability affects OpenSSL 1.0.1 to 1.0.1f and 1.0.2 to 1.0.2 beta versions. An attacker can use this vulnerability to unauthorized read sensitive information on the server, including encryption keys, etc.

How do you write unit tests in Go?How do you write unit tests in Go?Mar 21, 2025 pm 06:34 PM

The article discusses writing unit tests in Go, covering best practices, mocking techniques, and tools for efficient test management.

How do I write mock objects and stubs for testing in Go?How do I write mock objects and stubs for testing in Go?Mar 10, 2025 pm 05:38 PM

This article demonstrates creating mocks and stubs in Go for unit testing. It emphasizes using interfaces, provides examples of mock implementations, and discusses best practices like keeping mocks focused and using assertion libraries. The articl

How can I define custom type constraints for generics in Go?How can I define custom type constraints for generics in Go?Mar 10, 2025 pm 03:20 PM

This article explores Go's custom type constraints for generics. It details how interfaces define minimum type requirements for generic functions, improving type safety and code reusability. The article also discusses limitations and best practices

Explain the purpose of Go's reflect package. When would you use reflection? What are the performance implications?Explain the purpose of Go's reflect package. When would you use reflection? What are the performance implications?Mar 25, 2025 am 11:17 AM

The article discusses Go's reflect package, used for runtime manipulation of code, beneficial for serialization, generic programming, and more. It warns of performance costs like slower execution and higher memory use, advising judicious use and best

How can I use tracing tools to understand the execution flow of my Go applications?How can I use tracing tools to understand the execution flow of my Go applications?Mar 10, 2025 pm 05:36 PM

This article explores using tracing tools to analyze Go application execution flow. It discusses manual and automatic instrumentation techniques, comparing tools like Jaeger, Zipkin, and OpenTelemetry, and highlighting effective data visualization

How do you use table-driven tests in Go?How do you use table-driven tests in Go?Mar 21, 2025 pm 06:35 PM

The article discusses using table-driven tests in Go, a method that uses a table of test cases to test functions with multiple inputs and outcomes. It highlights benefits like improved readability, reduced duplication, scalability, consistency, and a

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
3 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. Best Graphic Settings
3 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. How to Fix Audio if You Can't Hear Anyone
3 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

SAP NetWeaver Server Adapter for Eclipse

SAP NetWeaver Server Adapter for Eclipse

Integrate Eclipse with SAP NetWeaver application server.

MinGW - Minimalist GNU for Windows

MinGW - Minimalist GNU for Windows

This project is in the process of being migrated to osdn.net/projects/mingw, you can continue to follow us there. MinGW: A native Windows port of the GNU Compiler Collection (GCC), freely distributable import libraries and header files for building native Windows applications; includes extensions to the MSVC runtime to support C99 functionality. All MinGW software can run on 64-bit Windows platforms.

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

WebStorm Mac version

WebStorm Mac version

Useful JavaScript development tools