How Can You Efficiently Detect File Changes on an NTFS Volume Using FSCTL_ENUM_USN_DATA?

Efficiently Detecting File Changes on an NTFS Volume Using FSCTL_ENUM_USN_DATA

Background

Existing backup methods, which check each file's archive bit, can become slow and inefficient for large file systems. This approach requires scanning all files, including temporary files, and may result in lengthy backup processes.

Alternative Approach using File System USN

A more efficient method involves using the File System USN (Update Sequence Number) change journal. Filesystem USN provides a record for each change made to the file system, including file creation, deletion, and modification.

How FSCTL_ENUM_USN_DATA Works

To detect changes on an NTFS volume, we can utilize the FSCTL_ENUM_USN_DATA control code. This control code:

• Enumerates all files on a volume, including only currently existing ones.

• Retrieves critical data for each file, including:

  • File flags
  • USN
  • File names
  • Parent file reference numbers

Implementing Change Detection

To detect changes:

1. Obtain File System USN Data: Use FSCTL_QUERY_USN_JOURNAL to get the system's maximum USN (maxusn).
2. Enumerate USN Records: Use a loop to iterate through USN records using FSCTL_ENUM_USN_DATA.
3. Identify Relevant Records: Check flags and compare USNs to detect created, deleted, or modified files.
4. Resolve Parent Paths: Match parent file reference numbers with file reference numbers of directories to obtain complete file paths.

Code Example in C

Here's a code snippet that demonstrates the approach:

<code class="c++">DWORDLONG nextid;
DWORDLONG filecount = 0;
DWORD starttick, endtick;

// Allocate memory for USN records
void * buffer = VirtualAlloc(NULL, BUFFER_SIZE, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE);

// Open volume handle
HANDLE drive = CreateFile(L"\\?\c:", GENERIC_READ, FILE_SHARE_DELETE | FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_ALWAYS, FILE_FLAG_NO_BUFFERING, NULL);

// Get volume USN journal data
USN_JOURNAL_DATA * journal = (USN_JOURNAL_DATA *)buffer;
if (!DeviceIoControl(drive, FSCTL_QUERY_USN_JOURNAL, NULL, 0, buffer, BUFFER_SIZE, &bytecount, NULL)) {
  (...)
}
maxusn = journal->MaxUsn;

MFT_ENUM_DATA mft_enum_data;
mft_enum_data.StartFileReferenceNumber = 0;
mft_enum_data.LowUsn = 0;
mft_enum_data.HighUsn = maxusn;

while (...) {
  if (!DeviceIoControl(drive, FSCTL_ENUM_USN_DATA, &mft_enum_data, sizeof(mft_enum_data), buffer, BUFFER_SIZE, &bytecount, NULL)) {
    (...)
  }

  nextid = *((DWORDLONG *)buffer);
  USN_RECORD * record = (USN_RECORD *)((USN *)buffer + 1);
  USN_RECORD * recordend = (USN_RECORD *)(((BYTE *)buffer) + bytecount);

  while (record < recordend) {
    filecount++;
    // Check flags and USNs to identify changes
    (...)
    record = (USN_RECORD *)(((BYTE *)record) + record->RecordLength);
  }
  mft_enum_data.StartFileReferenceNumber = nextid;
}</code>

Performance Considerations

The approach using FSCTL_ENUM_USN_DATA offers:

• Fast enumeration process: Capable of processing over 6000 records per second.
• Efficient filtering: Only relevant file change records are analyzed, eliminating overhead from temporary files.
• Potential limitations: Performance may vary on very large volumes, but it is generally more efficient than checking archive bits.

Additional Notes

• Replace MFT_ENUM_DATA with MFT_ENUM_DATA_V0 on Windows versions later than Windows 7.
• File reference numbers are printed as 32-bit, which is a mistake. In production code, it's recommended to use 64-bit values.

The above is the detailed content of How Can You Efficiently Detect File Changes on an NTFS Volume Using FSCTL_ENUM_USN_DATA?. For more information, please follow other related articles on the PHP Chinese website!