Issues with LevelDB database size reduction in Go (levigo)

php editor Youzi will introduce you to the size reduction problems and solutions that may be encountered when using the LevelDB database in Go. LevelDB is a high-performance key-value database, but when processing large amounts of data, the size of the database may grow rapidly and occupy a large amount of storage space. The article will discuss in detail how to solve this problem by using the levigo library and using compression algorithms to reduce the size of the database, thereby improving performance and saving storage space. Whether you are a beginner or an experienced developer, this article will help you.

Question content

Hello Stack Overflow community,

I am currently developing a Go program that utilizes LevelDB for data storage using the levigo package. My goal is to manage the database size efficiently, specifically deleting old records when available storage is low. However, I observed an unexpected behavior: the LevelDB database folder size did not decrease proportionally after deleting records.

Here is a simplified version of the code that reproduces the problem:

Save data code:

package main

import (
    "crypto/rand"
    "fmt"
    "log"

    "github.com/jmhodges/levigo"
)

func main() {
    // Specify the LevelDB options
    options := levigo.NewOptions()
    cache := levigo.NewLRUCache(5 << 20)
    options.SetCache(cache)
    options.SetCreateIfMissing(true)
    options.SetMaxOpenFiles(100)

    // Open or create the LevelDB database
    db, _ := levigo.Open("/tmp/mydatabase", options)
    defer db.Close()

    dataSize := 1024 * 1024 * 5 // 5MB
    randomData := make([]byte, dataSize)
    rand.Read(randomData)

    // Enqueue 5 pieces of data
    for i := 1; i <= 5; i++ {
        key := []byte(fmt.Sprintf("key%d", i))

        // Write the batch to the database
        if err := db.Put(levigo.NewWriteOptions(), key, randomData); err != nil {
            log.Fatal(err)
        }

        fmt.Printf("Enqueued: %s \n", key)
    }

    fmt.Println("Enqueue completed.")
}

Delete data code:

package main

import (
    "fmt"
    "log"

    "github.com/jmhodges/levigo"
)

func main() {
    // Specify the LevelDB options
    options := levigo.NewOptions()
    cache := levigo.NewLRUCache(5 << 20)
    options.SetCache(cache)
    options.SetCreateIfMissing(true)
    options.SetMaxOpenFiles(100)

    // Open or create the LevelDB database
    db, _ := levigo.Open("/tmp/mydatabase", options)
    defer db.Close()

    // Dequeue (remove) the 3 pieces of data
    for i := 1; i <= 3; i++ {
        key := []byte(fmt.Sprintf("key%d", i))

        // Create a WriteOptions for deleting from the database
        wo := levigo.NewWriteOptions()
        defer wo.Close()

        // Delete the key from the database
        if err := db.Delete(wo, key); err != nil {
            log.Fatal(err)
        }

        fmt.Printf("Dequeued: %s\n", key)
    }

    fmt.Println("Dequeue completed.")
}

After running the code and saving 5 items, the database folder size is 30MB. Later, when I ran the code to delete 3 items, the folder size was reduced to 26MB. Considering the amount of data removed, I expected the size to be reduced more significantly.

I have set LevelDB options such as cache size and file limits, but it seems I may have missed something during configuration or removal.

question:

1. What could cause the LevelDB database folder size not to decrease proportionally after deleting records?
2. Are there any other configurations or optimizations I should consider in order to manage the database size more efficiently?
3. Is there a specific way in levigo to compact the database to free up unused space?

Any insight or guidance on resolving this issue would be greatly appreciated. Thanks!

Workaround

By reading the question on this level DB repository I realized that I could add this to delete the row at the end of the loop db.CompactRange (levigo.Range{})
So the database will delete unused data and the total size of the database folder will be reduced.

The above is the detailed content of Issues with LevelDB database size reduction in Go (levigo). For more information, please follow other related articles on the PHP Chinese website!