Step 1: MongoDB Cursor
Here’s how we set up the cursor (reusing your snippet):
const cursor = userObject?.data?.serviceProviderName === 'ZYRO' ? zyroTransactionModel.find(query).cursor() : finoTransactionModel.find(query).cursor(); console.log("Cursor created successfully");
Step 2: Setting Up the ZIP File
Use the yazl library to stream CSV data into a ZIP file:
const yazl = require('yazl'); const zipfile = new yazl.ZipFile(); reply.raw.writeHead(200, { "Content-Type": "application/zip", "Content-Disposition": "attachment; filename=transactions.zip", }); zipfile.outputStream.pipe(reply.raw); const cleanup = async () => { console.log("Cleaning up resources..."); zipfile.end(); // Finalize ZIP await cursor.close(); }; reply.raw.on("close", cleanup); reply.raw.on("error", cleanup);
Step 3: Creating Dynamic CSV Streams
Generate CSV data dynamically and stream it into the ZIP file:
const createNewCSVStream = (headers) => { const csvStream = new Readable({ read() {} }); csvStream.push(headers.join(",") + "\n"); // Add headers return csvStream; }; const filteredHeaders = getHeaders(transactionDownloadFields, userObject?.state?.auth?.role); const currentCSVStream = createNewCSVStream(filteredHeaders); zipfile.addReadStream(currentCSVStream, "transactions_part_1.csv");
Step 4: Streaming MongoDB Data to CSV
Stream the data from MongoDB directly into the CSV:
cursor.on('data', (doc) => { const csvRow = filteredHeaders.map(header => doc[header.key] || '').join(','); currentCSVStream.push(csvRow + '\n'); // Write row }); cursor.on('end', () => { currentCSVStream.push(null); // End the stream zipfile.end(); // Finalize the ZIP });
Step 5: Processing Data from MongoDB Cursor
Stream documents from the MongoDB cursor, transform them as needed, and dynamically write rows to the CSV stream:
try { for await (const doc of cursor) { if (clientDisconnected) { console.log("Client disconnected. Stopping processing..."); break; } streamedCount++; rowCount++; let row = ""; const filteredHeaders = getHeaders( transactionDownloadFields, userObject?.state?.auth?.role ); for (let i = 0; i f.value === field.value); // Get the value from the transaction document let value = getValueFromTransaction(doc, field.value); // Apply transformation if the field has a transform function if (originalField?.transform) { value = originalField.transform(value); } // Enclose the value in double quotes value = value !== undefined ? `"${value}"` : '"N/A"'; row += (i > 0 ? "," : "") + value; } row += "\n"; currentCSVStream.push(row); // Check if the row count has reached the threshold for the current CSV file if (rowCount >= MAX_ROWS_PER_FILE) { console.log(`Threshold reached for file ${fileIndex - 1}. Starting new file...`); currentCSVStream.push(null); // End the current CSV stream currentCSVStream = createNewCSVStream(); // Start a new stream rowCount = 0; // Reset the row count } } // Finalize the current CSV stream if it has data if (currentCSVStream) { currentCSVStream.push(null); } // Finalize the ZIP file zipfile.end(); console.log(`Successfully streamed ${streamedCount} rows across ${fileIndex - 1} files.`); } catch (error) { console.error("Error during processing:", error); if (!headersSent) reply.status(500).send({ error: "Failed to generate ZIP file" }); } finally { // Cleanup: Close the MongoDB cursor await cursor.close().catch((err) => console.error("Error closing cursor:", err)); }
Summary
Document Iteration Using for await...of:
Streams documents one by one from the MongoDB cursor efficiently.
Enables real-time processing without loading all data into memory.
- Dynamic CSV Row Generation:
Constructs each row dynamically by iterating over filteredHeaders.
Applies transformations using a transform function, if defined in transactionDownloadFields.
Row Threshold and File Splitting:
Monitors the row count against the threshold (MAX_ROWS_PER_FILE).
Ends the current CSV stream and starts a new one when the threshold is reached.
- Error Handling:
Logs and sends an error response if an issue occurs during processing.
Ensures proper cleanup by closing the MongoDB cursor in the finally block.
- Finalizing Streams:
Pushes null to terminate the current CSV stream.
Completes the ZIP file once all rows are processed.
The above is the detailed content of From Storage to Stream: Delivering MongoDB Data Directly to Users. For more information, please follow other related articles on the PHP Chinese website!

Detailed explanation of JavaScript string replacement method and FAQ This article will explore two ways to replace string characters in JavaScript: internal JavaScript code and internal HTML for web pages. Replace string inside JavaScript code The most direct way is to use the replace() method: str = str.replace("find","replace"); This method replaces only the first match. To replace all matches, use a regular expression and add the global flag g: str = str.replace(/fi

Leverage jQuery for Effortless Web Page Layouts: 8 Essential Plugins jQuery simplifies web page layout significantly. This article highlights eight powerful jQuery plugins that streamline the process, particularly useful for manual website creation

So here you are, ready to learn all about this thing called AJAX. But, what exactly is it? The term AJAX refers to a loose grouping of technologies that are used to create dynamic, interactive web content. The term AJAX, originally coined by Jesse J

10 fun jQuery game plugins to make your website more attractive and enhance user stickiness! While Flash is still the best software for developing casual web games, jQuery can also create surprising effects, and while not comparable to pure action Flash games, in some cases you can also have unexpected fun in your browser. jQuery tic toe game The "Hello world" of game programming now has a jQuery version. Source code jQuery Crazy Word Composition Game This is a fill-in-the-blank game, and it can produce some weird results due to not knowing the context of the word. Source code jQuery mine sweeping game

Article discusses creating, publishing, and maintaining JavaScript libraries, focusing on planning, development, testing, documentation, and promotion strategies.

This tutorial demonstrates how to create a captivating parallax background effect using jQuery. We'll build a header banner with layered images that create a stunning visual depth. The updated plugin works with jQuery 1.6.4 and later. Download the

This tutorial demonstrates creating dynamic page boxes loaded via AJAX, enabling instant refresh without full page reloads. It leverages jQuery and JavaScript. Think of it as a custom Facebook-style content box loader. Key Concepts: AJAX and jQuery

This JavaScript library leverages the window.name property to manage session data without relying on cookies. It offers a robust solution for storing and retrieving session variables across browsers. The library provides three core methods: Session


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

VSCode Windows 64-bit Download
A free and powerful IDE editor launched by Microsoft

DVWA
Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software

Notepad++7.3.1
Easy-to-use and free code editor

SecLists
SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

SublimeText3 Mac version
God-level code editing software (SublimeText3)
