將大檔案上傳到雲端可能具有挑戰性 - 網路中斷、瀏覽器限制和龐大的檔案大小很容易破壞這個過程。 Amazon S3(簡單儲存服務)是一種可擴展、高速、基於 Web 的雲端儲存服務,專為資料和應用程式的線上備份和歸檔而設計。然而,將大檔案上傳到 S3 需要小心處理,以確保可靠性和效能。
AWS S3 的分段上傳:這是一個強大的解決方案,可以將大檔案分成更小的區塊,透過獨立處理每個部分甚至並行上傳部分來實現更快、更可靠的上傳。這種方法不僅克服了檔案大小限制(S3 需要對大於 5GB 的檔案進行分段上傳),而且還最大限度地降低了失敗的風險,使其非常適合需要無縫、穩健的檔案上傳的應用程式。
在本指南中,我們將詳細介紹客戶端分段上傳到 S3 的細節,向您展示為什麼它是處理大文件的明智選擇、如何安全地啟動和運行它以及需要注意哪些挑戰出去。我將提供逐步說明、程式碼範例和最佳實踐,以幫助您實現可靠的客戶端文件上傳解決方案。
準備好升級您的檔案上傳體驗了嗎?讓我們開始吧!
設計檔案上傳系統時,您有兩個主要選擇:透過伺服器上傳檔案(伺服器端)或直接從客戶端上傳檔案到 S3(客戶端)。每種方法都有其優點和缺點。
增強的安全性:所有上傳均由伺服器管理,確保 AWS 憑證的安全性。
更好的錯誤處理:伺服器可以更穩健地管理重試、日誌記錄和錯誤處理。
集中處理:檔案可以在儲存到 S3 之前在伺服器上進行驗證、處理或轉換。
更高的伺服器負載:大量上傳會消耗伺服器資源(CPU、記憶體、頻寬),這會影響效能並增加營運成本。
潛在瓶頸:在高上傳流量期間,伺服器可能成為單點故障或效能瓶頸,導致上傳緩慢或停機。
成本增加:在伺服器端處理上傳可能需要擴展基礎設施以處理尖峰負載,從而增加營運費用。
減少伺服器負載:檔案直接從使用者裝置傳送到 S3,釋放伺服器資源。
速度提高:由於繞過應用程式伺服器,使用者體驗更快的上傳速度。
成本效率:無需伺服器基礎設施來處理大型上傳,從而可能降低成本。
可擴充性:非常適合在不給後端伺服器造成壓力的情況下擴展檔案上傳。
安全風險:需要仔細處理 AWS 憑證和權限。必須安全地產生預簽名 URL,以防止未經授權的存取。
有限控制:伺服器端對上傳的監督較少;錯誤處理和重試通常在客戶端進行管理。
瀏覽器限制:瀏覽器有記憶體和 API 限制,這可能會阻礙處理非常大的檔案或影響低階裝置上的效能。
安全地實現客戶端上傳涉及前端應用程式和安全後端服務之間的協調。後端服務的主要作用是產生預簽名 URL,讓用戶端將檔案直接上傳到 S3,而無需暴露敏感的 AWS 憑證。
要有效實現客戶端上傳,您需要:
此架構確保敏感操作在後端安全處理,而前端則管理上傳過程。
預簽名 URL 允許客戶端直接與 S3 交互,執行上傳文件等操作,而無需在用戶端提供 AWS 憑證。它們是安全的,因為:
在伺服器上建立一個服務類,負責:
a.定義 S3 儲存桶和區域
b.安全地建立 AWS 憑證。
c.提供產生預簽名 URL 和管理分段上傳的方法。
// services/S3UploadService.js import { S3Client, CreateMultipartUploadCommand, CompleteMultipartUploadCommand, UploadPartCommand, AbortMultipartUploadCommand, PutObjectCommand, GetObjectCommand, DeleteObjectCommand, } from '@aws-sdk/client-s3'; import { getSignedUrl } from '@aws-sdk/s3-request-presigner'; // Import credential providers import { fromIni, fromInstanceMetadata, fromEnv, fromProcess, } from '@aws-sdk/credential-providers'; export class S3UploadService { constructor() { this.s3BucketName = process.env.S3_BUCKET_NAME; this.s3Region = process.env.S3_REGION; this.s3Client = new S3Client({ region: this.s3Region, credentials: this.getS3ClientCredentials(), }); } // Method to generate AWS credentials securely getS3ClientCredentials() { if (process.env.NODE_ENV === 'development') { // In development, use credentials from environment variables return fromEnv(); } else { // In production, use credentials from EC2 instance metadata or another secure method return fromInstanceMetadata(); } } // Generate a presigned URL for single-part upload (PUT), download (GET), or deletion (DELETE) async generatePresignedUrl(key, operation) { let command; switch (operation) { case 'PUT': command = new PutObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; case 'GET': command = new GetObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; case 'DELETE': command = new DeleteObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; default: throw new Error(`Invalid operation "${operation}"`); } // Generate presigned URL return await getSignedUrl(this.s3Client, command, { expiresIn: 3600 }); // Expires in 1 hour } // Methods for multipart upload async createMultipartUpload(key) { const command = new CreateMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, }); const response = await this.s3Client.send(command); return response.UploadId; } async generateUploadPartUrl(key, uploadId, partNumber) { const command = new UploadPartCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, PartNumber: partNumber, }); return await getSignedUrl(this.s3Client, command, { expiresIn: 3600 }); } async completeMultipartUpload(key, uploadId, parts) { const command = new CompleteMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts }, }); return await this.s3Client.send(command); } async abortMultipartUpload(key, uploadId) { const command = new AbortMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, }); return await this.s3Client.send(command); } }
注意:確保您的 AWS 憑證已安全管理。在生產中,建議使用附加到 EC2 執行個體或 ECS 任務的 IAM 角色,而不是硬編碼憑證或使用環境變數。
在後端建立 API 端點來處理來自前端的請求。這些端點將會利用 S3UploadService 來執行操作。
// controllers/S3UploadController.js import { S3UploadService } from '../services/S3UploadService'; const s3UploadService = new S3UploadService(); export const generatePresignedUrl = async (req, res, next) => { try { const { key, operation } = req.body; // key is the S3 object key (file identifier) const url = await s3UploadService.generatePresignedUrl(key, operation); res.status(200).json({ url }); } catch (error) { next(error); } }; export const initializeMultipartUpload = async (req, res, next) => { try { const { key } = req.body; const uploadId = await s3UploadService.createMultipartUpload(key); res.status(200).json({ uploadId }); } catch (error) { next(error); } }; export const generateUploadPartUrls = async (req, res, next) => { try { const { key, uploadId, parts } = req.body; // parts is the number of parts const urls = await Promise.all( [...Array(parts).keys()].map(async (index) => { const partNumber = index + 1; const url = await s3UploadService.generateUploadPartUrl(key, uploadId, partNumber); return { partNumber, url }; }) ); res.status(200).json({ urls }); } catch (error) { next(error); } }; export const completeMultipartUpload = async (req, res, next) => { try { const { key, uploadId, parts } = req.body; // parts is an array of { ETag, PartNumber } const result = await s3UploadService.completeMultipartUpload(key, uploadId, parts); res.status(200).json({ result }); } catch (error) { next(error); } }; export const abortMultipartUpload = async (req, res, next) => { try { const { key, uploadId } = req.body; await s3UploadService.abortMultipartUpload(key, uploadId); res.status(200).json({ message: 'Upload aborted' }); } catch (error) { next(error); } };
在 Express 應用程式或您使用的任何框架中設定這些端點的路由。
前端將處理選擇文件,根據文件大小決定是否執行單部分或分段上傳,並管理上傳過程。
一般來說,AWS 建議「當您的物件大小達到 100 MB 時,您應該考慮使用分段上傳,而不是在單一操作中上傳物件。」來源
// services/S3UploadService.js import { S3Client, CreateMultipartUploadCommand, CompleteMultipartUploadCommand, UploadPartCommand, AbortMultipartUploadCommand, PutObjectCommand, GetObjectCommand, DeleteObjectCommand, } from '@aws-sdk/client-s3'; import { getSignedUrl } from '@aws-sdk/s3-request-presigner'; // Import credential providers import { fromIni, fromInstanceMetadata, fromEnv, fromProcess, } from '@aws-sdk/credential-providers'; export class S3UploadService { constructor() { this.s3BucketName = process.env.S3_BUCKET_NAME; this.s3Region = process.env.S3_REGION; this.s3Client = new S3Client({ region: this.s3Region, credentials: this.getS3ClientCredentials(), }); } // Method to generate AWS credentials securely getS3ClientCredentials() { if (process.env.NODE_ENV === 'development') { // In development, use credentials from environment variables return fromEnv(); } else { // In production, use credentials from EC2 instance metadata or another secure method return fromInstanceMetadata(); } } // Generate a presigned URL for single-part upload (PUT), download (GET), or deletion (DELETE) async generatePresignedUrl(key, operation) { let command; switch (operation) { case 'PUT': command = new PutObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; case 'GET': command = new GetObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; case 'DELETE': command = new DeleteObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; default: throw new Error(`Invalid operation "${operation}"`); } // Generate presigned URL return await getSignedUrl(this.s3Client, command, { expiresIn: 3600 }); // Expires in 1 hour } // Methods for multipart upload async createMultipartUpload(key) { const command = new CreateMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, }); const response = await this.s3Client.send(command); return response.UploadId; } async generateUploadPartUrl(key, uploadId, partNumber) { const command = new UploadPartCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, PartNumber: partNumber, }); return await getSignedUrl(this.s3Client, command, { expiresIn: 3600 }); } async completeMultipartUpload(key, uploadId, parts) { const command = new CompleteMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts }, }); return await this.s3Client.send(command); } async abortMultipartUpload(key, uploadId) { const command = new AbortMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, }); return await this.s3Client.send(command); } }
// controllers/S3UploadController.js import { S3UploadService } from '../services/S3UploadService'; const s3UploadService = new S3UploadService(); export const generatePresignedUrl = async (req, res, next) => { try { const { key, operation } = req.body; // key is the S3 object key (file identifier) const url = await s3UploadService.generatePresignedUrl(key, operation); res.status(200).json({ url }); } catch (error) { next(error); } }; export const initializeMultipartUpload = async (req, res, next) => { try { const { key } = req.body; const uploadId = await s3UploadService.createMultipartUpload(key); res.status(200).json({ uploadId }); } catch (error) { next(error); } }; export const generateUploadPartUrls = async (req, res, next) => { try { const { key, uploadId, parts } = req.body; // parts is the number of parts const urls = await Promise.all( [...Array(parts).keys()].map(async (index) => { const partNumber = index + 1; const url = await s3UploadService.generateUploadPartUrl(key, uploadId, partNumber); return { partNumber, url }; }) ); res.status(200).json({ urls }); } catch (error) { next(error); } }; export const completeMultipartUpload = async (req, res, next) => { try { const { key, uploadId, parts } = req.body; // parts is an array of { ETag, PartNumber } const result = await s3UploadService.completeMultipartUpload(key, uploadId, parts); res.status(200).json({ result }); } catch (error) { next(error); } }; export const abortMultipartUpload = async (req, res, next) => { try { const { key, uploadId } = req.body; await s3UploadService.abortMultipartUpload(key, uploadId); res.status(200).json({ message: 'Upload aborted' }); } catch (error) { next(error); } };
雖然AWS S3 支援大小高達5 TiB(太字節)的對象,但由於瀏覽器限制和客戶端資源限制,直接從瀏覽器上傳如此大的檔案是不切實際的,而且通常是不可能的。處理非常大的檔案時,瀏覽器可能會崩潰或變得無回應,特別是需要在記憶體中處理它們時。
上傳大檔案會增加上傳過程中網路中斷或失敗的風險。實施穩健的重試策略對於增強使用者體驗並確保成功上傳至關重要。
不完整的分段上傳可能會累積在您的 S3 儲存桶中,消耗儲存空間並可能產生費用。
生命週期規則配置範例:
// services/S3UploadService.js import { S3Client, CreateMultipartUploadCommand, CompleteMultipartUploadCommand, UploadPartCommand, AbortMultipartUploadCommand, PutObjectCommand, GetObjectCommand, DeleteObjectCommand, } from '@aws-sdk/client-s3'; import { getSignedUrl } from '@aws-sdk/s3-request-presigner'; // Import credential providers import { fromIni, fromInstanceMetadata, fromEnv, fromProcess, } from '@aws-sdk/credential-providers'; export class S3UploadService { constructor() { this.s3BucketName = process.env.S3_BUCKET_NAME; this.s3Region = process.env.S3_REGION; this.s3Client = new S3Client({ region: this.s3Region, credentials: this.getS3ClientCredentials(), }); } // Method to generate AWS credentials securely getS3ClientCredentials() { if (process.env.NODE_ENV === 'development') { // In development, use credentials from environment variables return fromEnv(); } else { // In production, use credentials from EC2 instance metadata or another secure method return fromInstanceMetadata(); } } // Generate a presigned URL for single-part upload (PUT), download (GET), or deletion (DELETE) async generatePresignedUrl(key, operation) { let command; switch (operation) { case 'PUT': command = new PutObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; case 'GET': command = new GetObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; case 'DELETE': command = new DeleteObjectCommand({ Bucket: this.s3BucketName, Key: key, }); break; default: throw new Error(`Invalid operation "${operation}"`); } // Generate presigned URL return await getSignedUrl(this.s3Client, command, { expiresIn: 3600 }); // Expires in 1 hour } // Methods for multipart upload async createMultipartUpload(key) { const command = new CreateMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, }); const response = await this.s3Client.send(command); return response.UploadId; } async generateUploadPartUrl(key, uploadId, partNumber) { const command = new UploadPartCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, PartNumber: partNumber, }); return await getSignedUrl(this.s3Client, command, { expiresIn: 3600 }); } async completeMultipartUpload(key, uploadId, parts) { const command = new CompleteMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts }, }); return await this.s3Client.send(command); } async abortMultipartUpload(key, uploadId) { const command = new AbortMultipartUploadCommand({ Bucket: this.s3BucketName, Key: key, UploadId: uploadId, }); return await this.s3Client.send(command); } }
上傳大檔案可能會佔用大量資源,並可能導致瀏覽器主執行緒無回應,從而導致使用者體驗不佳。
在實作客戶端分段上傳時,瀏覽器相容性確實是一個問題。不同的瀏覽器可能對處理大文件上傳所需的 API 和功能有不同程度的支持,例如 *文件 API、Blob 切片、Web Workers 和網路請求處理* 。成功應對這些差異對於確保在所有支援的瀏覽器上獲得一致且可靠的使用者體驗至關重要。
透過使用預簽名 URL 和分段上傳實現客戶端上傳,您可以有效地直接處理任意大小的檔案上傳到 S3,從而減少伺服器負載並提高效能。請記住,透過安全地管理 AWS 憑證並限制預簽 URL 的權限和生命週期,將安全性放在首位。
本指南提供了使用 AWS S3、AWS SDK for JavaScript 和預簽名 URL 設定安全且可擴充的檔案上傳系統的逐步方法。透過提供的程式碼範例和最佳實踐,您就可以很好地增強應用程式的檔案上傳功能。
以上是優化大檔案上傳:安全地將客戶端分段上傳到 AWS S3的詳細內容。更多資訊請關注PHP中文網其他相關文章!