Step 1: MongoDB Cursor
Here’s how we set up the cursor (reusing your snippet):
const cursor = userObject?.data?.serviceProviderName === 'ZYRO' ? zyroTransactionModel.find(query).cursor() : finoTransactionModel.find(query).cursor(); console.log("Cursor created successfully");
Step 2: Setting Up the ZIP File
Use the yazl library to stream CSV data into a ZIP file:
const yazl = require('yazl'); const zipfile = new yazl.ZipFile(); reply.raw.writeHead(200, { "Content-Type": "application/zip", "Content-Disposition": "attachment; filename=transactions.zip", }); zipfile.outputStream.pipe(reply.raw); const cleanup = async () => { console.log("Cleaning up resources..."); zipfile.end(); // Finalize ZIP await cursor.close(); }; reply.raw.on("close", cleanup); reply.raw.on("error", cleanup);
Step 3: Creating Dynamic CSV Streams
Generate CSV data dynamically and stream it into the ZIP file:
const createNewCSVStream = (headers) => { const csvStream = new Readable({ read() {} }); csvStream.push(headers.join(",") + "\n"); // Add headers return csvStream; }; const filteredHeaders = getHeaders(transactionDownloadFields, userObject?.state?.auth?.role); const currentCSVStream = createNewCSVStream(filteredHeaders); zipfile.addReadStream(currentCSVStream, "transactions_part_1.csv");
Step 4: Streaming MongoDB Data to CSV
Stream the data from MongoDB directly into the CSV:
cursor.on('data', (doc) => { const csvRow = filteredHeaders.map(header => doc[header.key] || '').join(','); currentCSVStream.push(csvRow + '\n'); // Write row }); cursor.on('end', () => { currentCSVStream.push(null); // End the stream zipfile.end(); // Finalize the ZIP });
Step 5: Processing Data from MongoDB Cursor
Stream documents from the MongoDB cursor, transform them as needed, and dynamically write rows to the CSV stream:
try { for await (const doc of cursor) { if (clientDisconnected) { console.log("Client disconnected. Stopping processing..."); break; } streamedCount++; rowCount++; let row = ""; const filteredHeaders = getHeaders( transactionDownloadFields, userObject?.state?.auth?.role ); for (let i = 0; i f.value === field.value); // Get the value from the transaction document let value = getValueFromTransaction(doc, field.value); // Apply transformation if the field has a transform function if (originalField?.transform) { value = originalField.transform(value); } // Enclose the value in double quotes value = value !== undefined ? `"${value}"` : '"N/A"'; row += (i > 0 ? "," : "") + value; } row += "\n"; currentCSVStream.push(row); // Check if the row count has reached the threshold for the current CSV file if (rowCount >= MAX_ROWS_PER_FILE) { console.log(`Threshold reached for file ${fileIndex - 1}. Starting new file...`); currentCSVStream.push(null); // End the current CSV stream currentCSVStream = createNewCSVStream(); // Start a new stream rowCount = 0; // Reset the row count } } // Finalize the current CSV stream if it has data if (currentCSVStream) { currentCSVStream.push(null); } // Finalize the ZIP file zipfile.end(); console.log(`Successfully streamed ${streamedCount} rows across ${fileIndex - 1} files.`); } catch (error) { console.error("Error during processing:", error); if (!headersSent) reply.status(500).send({ error: "Failed to generate ZIP file" }); } finally { // Cleanup: Close the MongoDB cursor await cursor.close().catch((err) => console.error("Error closing cursor:", err)); }
Summary
Document Iteration Using for await...of:
Streams documents one by one from the MongoDB cursor efficiently.
Enables real-time processing without loading all data into memory.
- Dynamic CSV Row Generation:
Constructs each row dynamically by iterating over filteredHeaders.
Applies transformations using a transform function, if defined in transactionDownloadFields.
Row Threshold and File Splitting:
Monitors the row count against the threshold (MAX_ROWS_PER_FILE).
Ends the current CSV stream and starts a new one when the threshold is reached.
- Error Handling:
Logs and sends an error response if an issue occurs during processing.
Ensures proper cleanup by closing the MongoDB cursor in the finally block.
- Finalizing Streams:
Pushes null to terminate the current CSV stream.
Completes the ZIP file once all rows are processed.
The above is the detailed content of From Storage to Stream: Delivering MongoDB Data Directly to Users. For more information, please follow other related articles on the PHP Chinese website!

Detailed explanation of JavaScript string replacement method and FAQ This article will explore two ways to replace string characters in JavaScript: internal JavaScript code and internal HTML for web pages. Replace string inside JavaScript code The most direct way is to use the replace() method: str = str.replace("find","replace"); This method replaces only the first match. To replace all matches, use a regular expression and add the global flag g: str = str.replace(/fi

This tutorial shows you how to integrate a custom Google Search API into your blog or website, offering a more refined search experience than standard WordPress theme search functions. It's surprisingly easy! You'll be able to restrict searches to y

Leverage jQuery for Effortless Web Page Layouts: 8 Essential Plugins jQuery simplifies web page layout significantly. This article highlights eight powerful jQuery plugins that streamline the process, particularly useful for manual website creation

So here you are, ready to learn all about this thing called AJAX. But, what exactly is it? The term AJAX refers to a loose grouping of technologies that are used to create dynamic, interactive web content. The term AJAX, originally coined by Jesse J

Core points This in JavaScript usually refers to an object that "owns" the method, but it depends on how the function is called. When there is no current object, this refers to the global object. In a web browser, it is represented by window. When calling a function, this maintains the global object; but when calling an object constructor or any of its methods, this refers to an instance of the object. You can change the context of this using methods such as call(), apply(), and bind(). These methods call the function using the given this value and parameters. JavaScript is an excellent programming language. A few years ago, this sentence was

This post compiles helpful cheat sheets, reference guides, quick recipes, and code snippets for Android, Blackberry, and iPhone app development. No developer should be without them! Touch Gesture Reference Guide (PDF) A valuable resource for desig

jQuery is a great JavaScript framework. However, as with any library, sometimes it’s necessary to get under the hood to discover what’s going on. Perhaps it’s because you’re tracing a bug or are just curious about how jQuery achieves a particular UI

Article discusses creating, publishing, and maintaining JavaScript libraries, focusing on planning, development, testing, documentation, and promotion strategies.


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Atom editor mac version download
The most popular open source editor

mPDF
mPDF is a PHP library that can generate PDF files from UTF-8 encoded HTML. The original author, Ian Back, wrote mPDF to output PDF files "on the fly" from his website and handle different languages. It is slower than original scripts like HTML2FPDF and produces larger files when using Unicode fonts, but supports CSS styles etc. and has a lot of enhancements. Supports almost all languages, including RTL (Arabic and Hebrew) and CJK (Chinese, Japanese and Korean). Supports nested block-level elements (such as P, DIV),

SublimeText3 Linux new version
SublimeText3 Linux latest version

VSCode Windows 64-bit Download
A free and powerful IDE editor launched by Microsoft

ZendStudio 13.5.1 Mac
Powerful PHP integrated development environment
