Use new io.FS to fs.WalkDir and list files across file system types

php editor Strawberry recommends using the new io.FS to operate the file system, especially when using fs.WalkDir and listing files across file system types. This new feature can handle file operations more flexibly, allowing users to more easily obtain the file information they need. Whether in the development or maintenance process, this function can greatly improve efficiency and reduce tedious operations. If you want to know more about how to use this function, you may wish to continue reading and we will introduce you to the detailed steps and precautions.

Question content

I am using the new io.FS abstraction to traverse the file system and read the first 128 bytes of each file that matches our internal file extension.

These files are located in local file systems and archive files etc. (ZIP and Tar I thunk).

I'm using fs.WalkDir, passing in fs.FS (os.DIR and fstest.MapFS in my tests). When walking, I return a set of "files" (actually they are *.pzix and *.pzi files, which are our proprietary formats). I can't find a suitable way to use the FS interface to get some information about the file I'm working on.

I want:

• Get file name
• Get file size
• Get openfile method

I find the interfaces from Java/C# in Go a bit confusing. I wish to operate on the abstraction, but I don't know how to get the other implementations of the file itself (e.g. the file interface has Stat() and read).

The easiest thing I've found is to store the path and filename in an array, then when I iterate over the array, determine if it's os.Dir or fstest.MapFS, but this seems rather counterintuitive:

func collectFiles(f fs.FS, root string) []string {
 var files []string
 fs.WalkDir(f, ".", func(p string, d fs.DirEntry, err error) error {
  if !d.IsDir() { // we also check a few other things in the filename here
   f = filepath.Abs(path.Join(root, p))
   files = append(files, f)
  }
 }
 return files
}

This gives me:

root = "m://" // mapfs
files = { "m://id-198271.pzi", "m://id-7125-092581.pzix"}

Is there a smarter way for me to handle the abstraction without doing these things? Because after returning the array, I have to "open" the file, read the first 128 bytes (the signature) and hash the rest of the file to make sure it's "valid".

Edit: To clarify, the collectFiles method is creating our main file hit list to be processed in another method. I want to pass local system files, zip files, and tar files into the method so that it can iterate over the files in the archive and add them to an array.

Wish there was a File interface that I could store in an array instead of a string so that subsequent callers could do f.open() without knowing what the underlying was.

Solution

p is the name in the file system.

Get the size by calling fs.Stat(f, p)

Use

f.Open(p) to open the file

Example:

f := os.DirFS("/etc")
fs.WalkDir(f, ".", func(p string, d fs.DirEntry, err error) error {
    if !d.IsDir() {
        st, _ := fs.Stat(f, p)
        r, _ := f.Open(p)
        defer r.Close()

        // Read prefix
        var buf [md5.Size]byte
        n, _ := io.ReadFull(r, buf[:])

        // Hash remainder
        h := md5.New()
        _, _ = io.Copy(h, r)
        s := h.Sum(nil)

        fmt.Printf("%s %d %x %x\n", p, st.Size(), buf[:n], s)
    }
    return nil
})

For the sake of brevity, this example ignores errors. Don't do this in real code.

https://www.php.cn/link/b599e8250e4481aaa405a715419c8179

The above is the detailed content of Use new io.FS to fs.WalkDir and list files across file system types. For more information, please follow other related articles on the PHP Chinese website!