How to download an object from AWS S3 into memory and send it via request in Go?

Question content

Can someone help me resolve this error I'm getting when trying to download a file from s3?

So I want to create a storage service api via go gin, and I want a route that uses an s3 object key to download an object and then send this object back to the client.

cfg,_ := config.loaddefaultconfig(context.todo())

// create an amazon s3 service client
s3client := s3.newfromconfig(cfg)
downloader := manager.newdownloader(s3client)

router.post("/download-s3", func(ctx *gin.context) {
        var data map[string]string

        if err := ctx.shouldbindjson(&data); err != nil {
            log.println("error: bind error")
            ctx.json(http.statusbadrequest, gin.h{"error": err.error()})
            return
        }

        log.println("body request data (filekey): ", data["filekey"])

        filekey := data["filekey"]

        /// buffer read
        buf := make([]byte, 100)
        // wrap with aws.writeatbuffer
        w := manager.newwriteatbuffer(buf)
        // download file into the memory
        numbytesdownloaded, err := downloader.download(ctx, w, &s3.getobjectinput{
            bucket: aws.string(bucketname),
            key:    aws.string(filekey),
        })
        if err != nil {
            log.println("error: download error")
            ctx.json(http.statusbadrequest, gin.h{"error": err.error()})
            return
        }

        ctx.json(http.statusaccepted,
            gin.h{
                "message":            fmt.sprintf("'%s' downloaded!", "test.jpg"),
                "numbytesdownloaded": numbytesdownloaded,
            },
        )

        ctx.data(http.statusok, "application/octet-stream", w.bytes())

    })

The above code gives me this error: Operation error s3: getobject, https response error statuscode: 403, requestid: 8khyw3rcs2za7d95, hostid: kolo gy/dtzrrovswsb1bxinln8w xdl0lnmysuwjnhupmbvk4itdfp mq2xuo8ehasnx/fai4i=, api error accessdenied: Access denied phpcnendc phpcn

The request body is like this:

{
    "filekey": "https://xxx-xxx.s3.ap-southeast-1.amazonaws.com/agencies/photo/06%3A03%3A2023_17%3A42%3A42_gggi_db_er.png"
}

I have all these aws_s3_bucket_name, aws_region, aws_access_key_id, aws_secret_access_key stored in .env files, and I think I have them set correctly because I have another route that uploads files to s3 (which works great).

Correct answer

Your filekey is this url:

"https://xxx-xxx.s3.ap-southeast-1.amazonaws.com/agencies/photo/06%3A03%3A2023_17%3A42%3A42_gggi_db_er.png"

This can only be used from an http client, such as a web browser or curl.

Specifically, it is not a valid s3 object key. When using getobject or similar s3 api it needs to be a valid key such as agcies/photo/xyz.png and should not be url encoded.

The above is the detailed content of How to download an object from AWS S3 into memory and send it via request in Go?. For more information, please follow other related articles on the PHP Chinese website!