Home > Article > Web Front-end > Optimizing Large File Uploads: Secure Client-Side Multipart Uploads to AWS S3
Uploading large files to the cloud can be challenging — network interruptions, browser limitations, and huge file sizes can easily disrupt the process. Amazon S3 (Simple Storage Service) is a scalable, high-speed, web-based cloud storage service designed for online backup and archiving of data and applications. However, uploading large files to S3 requires careful handling to ensure reliability and performance.
Enter AWS S3’s multipart upload: a powerful solution that breaks big files into smaller chunks, enabling faster, more reliable uploads by tackling each part independently and even uploading parts in parallel. This method not only overcomes file size limits (S3 requires multipart upload for files larger than 5GB) but also minimizes the risk of failure, making it a perfect fit for applications needing seamless, robust file uploads.
In this guide, we’ll unpack the ins and outs of client-side multipart uploads to S3, showing you why it’s the smart choice for handling large files, how to get it up and running securely, and what challenges to watch out for. I’ll provide step-by-step instructions, code examples, and best practices to help you implement a reliable client-side file upload solution.
Ready to upgrade your file upload experience? Let’s dive in!
When designing a file upload system, you have two primary options: uploading files through your server (server-side) or uploading files directly from the client to S3 (client-side). Each approach has its pros and cons.
Enhanced Security: All uploads are managed by the server, keeping AWS credentials secure.
Better Error Handling: Servers can manage retries, logging, and error handling more robustly.
Centralized Processing: Files can be validated, processed, or converted on the server before storing in S3.
Higher Server Load: Large uploads consume server resources (CPU, memory, bandwidth), which can impact performance and increase operational costs.
Potential Bottlenecks: The server can become a single point of failure or a performance bottleneck during high upload traffic, leading to slow uploads or downtime.
Increased Costs: Handling uploads server-side may require scaling your infrastructure to handle peak loads, raising operational expenses.
Reduced Server Load: Files are sent directly from the user’s device to S3, freeing up server resources.
Improved Speed: Users experience faster uploads since they bypass the application server.
Cost Efficiency: Eliminates the need for server infrastructure to handle large uploads, potentially lowering costs.
Scalability: Ideal for scaling file uploads without stressing backend servers.
Security Risks: Requires careful handling of AWS credentials and permissions. Presigned URLs must be securely generated to prevent unauthorized access.
Limited Control: Less server-side oversight over uploads; error handling and retries are often managed on the client.
Browser Constraints: Browsers have memory and API limitations, which can hinder handling of very large files or affect performance on lower-end devices.
Implementing client-side uploads securely involves coordinating between your frontend application and a secure backend service. The backend service’s primary role is to generate presigned URLs, allowing the client to upload files directly to S3 without exposing sensitive AWS credentials.
To implement client-side uploads effectively, you need:
This architecture ensures that sensitive operations are handled securely on the backend, while the frontend manages the upload process.
Presigned URLs allow clients to interact with S3 directly, performing operations like uploading files without requiring AWS credentials on the client side. They are secure because:
Create a service class on your server responsible for:
a. Defining the S3 bucket and region
b. Establishing AWS credentials securely.
c. Providing methods to generate presigned URLs and manage multipart uploads.
// services/S3UploadService.js import { S3Client, CreateMultipartUploadCommand, CompleteMultipartUploadCommand, UploadPartCommand, AbortMultipartUploadCommand, PutObjectCommand, GetObjectCommand, DeleteObjectCommand, } from '@aws-sdk/client-s3'; import { getSignedUrl } from '@aws-sdk/s3-request-presigner'; // Import credential providers import { fromIni, fromInstanceMetadata, fromEnv, fromProcess, } from '@aws-sdk/credential-providers'; export class S3UploadService { constructor() { this.s3BucketName = process.env.S3_BUCKET_NAME; this.s3Region = process.env.S3_REGION; this.s3Client = new S3Client({ region: this.s3Region, credentials: this.getS3ClientCredentials(), }); } // Method to generate AWS credentials securely getS3ClientCredentials() { if (process.env.NODE_ENV === 'development') { // In development, use credentials from environment variables return fromEnv(); } else { // In production, use credentials from EC2 instance metadata or another secure method return fromInstanceMetadata(); } } // Generate a presigned URL for single-part upload (PUT), download (GET), or deletion (DELETE) async generatePresignedUrl(key, operation) { let command; switch (operation) { case 'PUT': command = new PutObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; case 'GET': command = new GetObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; case 'DELETE': command = new DeleteObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; default: throw new Error(`Invalid operation "${operation}"`); } // Generate presigned URL return await getSignedUrl(this.s3Client, command, { expiresIn: 3600 }); // Expires in 1 hour } // Methods for multipart upload async createMultipartUpload(key) { const command = new CreateMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, }); const response = await this.s3Client.send(command); return response.UploadId; } async generateUploadPartUrl(key, uploadId, partNumber) { const command = new UploadPartCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, PartNumber: partNumber, }); return await getSignedUrl(this.s3Client, command, { expiresIn: 3600 }); } async completeMultipartUpload(key, uploadId, parts) { const command = new CompleteMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts }, }); return await this.s3Client.send(command); } async abortMultipartUpload(key, uploadId) { const command = new AbortMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, }); return await this.s3Client.send(command); } }
Note: Ensure that your AWS credentials are securely managed. In production, it’s recommended to use IAM roles attached to your EC2 instances or ECS tasks, rather than hardcoding credentials or using environment variables.
Create API endpoints in your backend to handle requests from the frontend. These endpoints will utilize the S3UploadService to perform actions.
// controllers/S3UploadController.js import { S3UploadService } from '../services/S3UploadService'; const s3UploadService = new S3UploadService(); export const generatePresignedUrl = async (req, res, next) => { try { const { key, operation } = req.body; // key is the S3 object key (file identifier) const url = await s3UploadService.generatePresignedUrl(key, operation); res.status(200).json({ url }); } catch (error) { next(error); } }; export const initializeMultipartUpload = async (req, res, next) => { try { const { key } = req.body; const uploadId = await s3UploadService.createMultipartUpload(key); res.status(200).json({ uploadId }); } catch (error) { next(error); } }; export const generateUploadPartUrls = async (req, res, next) => { try { const { key, uploadId, parts } = req.body; // parts is the number of parts const urls = await Promise.all( [...Array(parts).keys()].map(async (index) => { const partNumber = index + 1; const url = await s3UploadService.generateUploadPartUrl(key, uploadId, partNumber); return { partNumber, url }; }) ); res.status(200).json({ urls }); } catch (error) { next(error); } }; export const completeMultipartUpload = async (req, res, next) => { try { const { key, uploadId, parts } = req.body; // parts is an array of { ETag, PartNumber } const result = await s3UploadService.completeMultipartUpload(key, uploadId, parts); res.status(200).json({ result }); } catch (error) { next(error); } }; export const abortMultipartUpload = async (req, res, next) => { try { const { key, uploadId } = req.body; await s3UploadService.abortMultipartUpload(key, uploadId); res.status(200).json({ message: 'Upload aborted' }); } catch (error) { next(error); } };
Set up the routes for these endpoints in your Express app or whichever framework you’re using.
The frontend will handle selecting files, deciding whether to perform a single-part or multipart upload based on file size, and managing the upload process.
In general, AWS recommends "when your object size reaches 100 MB, you should consider using multipart uploads instead of uploading the object in a single operation." Source
// services/S3UploadService.js import { S3Client, CreateMultipartUploadCommand, CompleteMultipartUploadCommand, UploadPartCommand, AbortMultipartUploadCommand, PutObjectCommand, GetObjectCommand, DeleteObjectCommand, } from '@aws-sdk/client-s3'; import { getSignedUrl } from '@aws-sdk/s3-request-presigner'; // Import credential providers import { fromIni, fromInstanceMetadata, fromEnv, fromProcess, } from '@aws-sdk/credential-providers'; export class S3UploadService { constructor() { this.s3BucketName = process.env.S3_BUCKET_NAME; this.s3Region = process.env.S3_REGION; this.s3Client = new S3Client({ region: this.s3Region, credentials: this.getS3ClientCredentials(), }); } // Method to generate AWS credentials securely getS3ClientCredentials() { if (process.env.NODE_ENV === 'development') { // In development, use credentials from environment variables return fromEnv(); } else { // In production, use credentials from EC2 instance metadata or another secure method return fromInstanceMetadata(); } } // Generate a presigned URL for single-part upload (PUT), download (GET), or deletion (DELETE) async generatePresignedUrl(key, operation) { let command; switch (operation) { case 'PUT': command = new PutObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; case 'GET': command = new GetObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; case 'DELETE': command = new DeleteObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; default: throw new Error(`Invalid operation "${operation}"`); } // Generate presigned URL return await getSignedUrl(this.s3Client, command, { expiresIn: 3600 }); // Expires in 1 hour } // Methods for multipart upload async createMultipartUpload(key) { const command = new CreateMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, }); const response = await this.s3Client.send(command); return response.UploadId; } async generateUploadPartUrl(key, uploadId, partNumber) { const command = new UploadPartCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, PartNumber: partNumber, }); return await getSignedUrl(this.s3Client, command, { expiresIn: 3600 }); } async completeMultipartUpload(key, uploadId, parts) { const command = new CompleteMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts }, }); return await this.s3Client.send(command); } async abortMultipartUpload(key, uploadId) { const command = new AbortMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, }); return await this.s3Client.send(command); } }
// controllers/S3UploadController.js import { S3UploadService } from '../services/S3UploadService'; const s3UploadService = new S3UploadService(); export const generatePresignedUrl = async (req, res, next) => { try { const { key, operation } = req.body; // key is the S3 object key (file identifier) const url = await s3UploadService.generatePresignedUrl(key, operation); res.status(200).json({ url }); } catch (error) { next(error); } }; export const initializeMultipartUpload = async (req, res, next) => { try { const { key } = req.body; const uploadId = await s3UploadService.createMultipartUpload(key); res.status(200).json({ uploadId }); } catch (error) { next(error); } }; export const generateUploadPartUrls = async (req, res, next) => { try { const { key, uploadId, parts } = req.body; // parts is the number of parts const urls = await Promise.all( [...Array(parts).keys()].map(async (index) => { const partNumber = index + 1; const url = await s3UploadService.generateUploadPartUrl(key, uploadId, partNumber); return { partNumber, url }; }) ); res.status(200).json({ urls }); } catch (error) { next(error); } }; export const completeMultipartUpload = async (req, res, next) => { try { const { key, uploadId, parts } = req.body; // parts is an array of { ETag, PartNumber } const result = await s3UploadService.completeMultipartUpload(key, uploadId, parts); res.status(200).json({ result }); } catch (error) { next(error); } }; export const abortMultipartUpload = async (req, res, next) => { try { const { key, uploadId } = req.body; await s3UploadService.abortMultipartUpload(key, uploadId); res.status(200).json({ message: 'Upload aborted' }); } catch (error) { next(error); } };
While AWS S3 supports objects up to 5 TiB (terabytes) in size, uploading such massive files directly from a browser is impractical and often impossible due to browser limitations and client-side resource constraints. Browsers can crash or become unresponsive when handling extremely large files, especially if they need to be processed in memory.
Uploading large files increases the risk of network interruptions or failures during the upload process. Implementing a robust retry strategy is crucial to enhance the user experience and ensure successful uploads.
Incomplete multipart uploads can accumulate in your S3 bucket, consuming storage space and potentially incurring costs.
Example Lifecycle Rule Configuration:
// services/S3UploadService.js import { S3Client, CreateMultipartUploadCommand, CompleteMultipartUploadCommand, UploadPartCommand, AbortMultipartUploadCommand, PutObjectCommand, GetObjectCommand, DeleteObjectCommand, } from '@aws-sdk/client-s3'; import { getSignedUrl } from '@aws-sdk/s3-request-presigner'; // Import credential providers import { fromIni, fromInstanceMetadata, fromEnv, fromProcess, } from '@aws-sdk/credential-providers'; export class S3UploadService { constructor() { this.s3BucketName = process.env.S3_BUCKET_NAME; this.s3Region = process.env.S3_REGION; this.s3Client = new S3Client({ region: this.s3Region, credentials: this.getS3ClientCredentials(), }); } // Method to generate AWS credentials securely getS3ClientCredentials() { if (process.env.NODE_ENV === 'development') { // In development, use credentials from environment variables return fromEnv(); } else { // In production, use credentials from EC2 instance metadata or another secure method return fromInstanceMetadata(); } } // Generate a presigned URL for single-part upload (PUT), download (GET), or deletion (DELETE) async generatePresignedUrl(key, operation) { let command; switch (operation) { case 'PUT': command = new PutObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; case 'GET': command = new GetObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; case 'DELETE': command = new DeleteObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; default: throw new Error(`Invalid operation "${operation}"`); } // Generate presigned URL return await getSignedUrl(this.s3Client, command, { expiresIn: 3600 }); // Expires in 1 hour } // Methods for multipart upload async createMultipartUpload(key) { const command = new CreateMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, }); const response = await this.s3Client.send(command); return response.UploadId; } async generateUploadPartUrl(key, uploadId, partNumber) { const command = new UploadPartCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, PartNumber: partNumber, }); return await getSignedUrl(this.s3Client, command, { expiresIn: 3600 }); } async completeMultipartUpload(key, uploadId, parts) { const command = new CompleteMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts }, }); return await this.s3Client.send(command); } async abortMultipartUpload(key, uploadId) { const command = new AbortMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, }); return await this.s3Client.send(command); } }
Uploading large files can be resource-intensive and may cause the browser’s main thread to become unresponsive, leading to a poor user experience.
When implementing client-side multipart uploads, browser compatibility is indeed a concern. Different browsers may have varying levels of support for the APIs and features required for handling large file uploads, such as the *File API, Blob slicing, Web Workers, and network request handling*. Navigating these differences successfully is crucial to ensure a consistent and reliable user experience across all supported browsers.
By implementing client-side uploads with presigned URLs and multipart upload, you can efficiently handle file uploads of any size directly to S3, reducing server load and improving performance. Remember to keep security at the forefront by securely managing AWS credentials and limiting the permissions and lifespan of presigned URLs.
This guide provided a step-by-step approach to setting up a secure and scalable file upload system using AWS S3, the AWS SDK for JavaScript, and presigned URLs. With the provided code examples and best practices, you’re well on your way to enhancing your application’s file upload capabilities.
The above is the detailed content of Optimizing Large File Uploads: Secure Client-Side Multipart Uploads to AWS S3. For more information, please follow other related articles on the PHP Chinese website!