AWS

Check incomplete multipart uploads if AWS S3 storage size is abnormally big huge compared to the origin storage

CK CK 3788 
Created at
Updated at  
11,598 0 0

You should check incomplete multipart uploads if your Metrics in the S3 Management Console shows larger size than you actually use. 
 

In my case, I found the total size of S3 bucket was 180TB, but actually size I uploaded based on the origin storage was 40TB. I tried to compare all the files one by one, but haven't found any difference. 

I learned AWS S3 keeps incomplete multipart uploads constantly and that won't be deleted as long as user does not make any actions. This is something unusual architecture I haven't expected, so I paid for that incomplete multipart uploads, because people may expect the incomplete multi uploads would be deleted automatically as long as that parts are not integrated for very long time.

  • I found an article that AWS will charge the cost for the incomplete multipart uploads as long as I do not call Abort for the on-going uploads. https://docs.amazonaws.cn/en_us/AmazonS3/latest/userguide/abort-mpu.html

  • The automatic discovery will be essential, because you can not know the previous handle in case of abnormal termination or something similar to that.

$s3 = new S3Client([
    'version' => 'latest',
    'region'  => 'us-east-1'
]);
.
.
.
abnormal termination
  • One of my complaint is that most of user does not know what happen until they pay more than they expected. That is what user should take care and such an information is not easy to find on AWS Web Site.

 

Here's the action you can do, so you do not need to waste your money for the incomplete multipart uploads.

 

The action user should do is "Creating lifecycle rule" in the S3 console.

 

What you should do is select your S3 Bucket and you will see the menu like below:

Check incomplete multipart uploads if AWS S3 storage size is abnormally big huge compared to the origin storage

Once you "Create lifecycle rule", you will need to put/select options like below:

Check incomplete multipart uploads if AWS S3 storage size is abnormally big huge compared to the origin storage

Check incomplete multipart uploads if AWS S3 storage size is abnormally big huge compared to the origin storage

If everything goes fine, you will see the summary like below:

Check incomplete multipart uploads if AWS S3 storage size is abnormally big huge compared to the origin storage

The final action you should do is click "Create rule"

Check incomplete multipart uploads if AWS S3 storage size is abnormally big huge compared to the origin storage

 

Once you setup the above rule, your rule will be executed in 48 hours and its result will be like below:

 

Check incomplete multipart uploads if AWS S3 storage size is abnormally big huge compared to the origin storage

Tags AWS AWS S3 Incomplete Multipart Uploads AWS S3 Multipart Uploads AWS S3 Optimization Multipart Uploads S3 Management Console Facebook X
Comments 0
Similar posts
  1. How to Set Up and Deploy Containerized Applications with AWS ECS
    10,673
  2. A Python script to create an EC2 Spot instance using boto3
    1,602
  3. The Ultimate Data Engineering Roadmap
    1,072
  4. Where to get AWS Access Key ?
    11,262
  1. A Python script to create an EC2 Spot instance using boto3
    1,602
  2. A Python/PHP code to deregister anormalous instaces in Amazon ECS
    1,857
  3. Where to get AWS API Key at AWS Management Console?
    1,067
  4. Understanding Amazon Bedrock Agent for AI-based ChatBot
    644
  5. Media sharing service architecture in AWS
    10,366
  6. Enterprise Data Lake Architecture based on AWS
    10,405
  7. "502 Bad Gateway" Error easy to happen at the combination of nginx and node.js in AWS Elastic Beanstalk
    10,487
  8. How to Set Up and Deploy Containerized Applications with AWS ECS
    10,673
  9. Power of Containerized Deployment with AWS ECS
    10,597
  10. FAISS Already Does ANN... So Why Do Top Teams Still Pay for SageMaker?
    309
  11. Recommended Vector DB Structure for Amazon Bedrock Integration
    389
  12. Product Marketing & Recommendation Architecture in AWS for e-commerce website
    10,736
  13. What cause a 502 bad gateway error in AWS environment?
    444
  14. Building an AWS SAM(Serverless Application Model) Project
    898
  15. Teraform from the scratch
    1,356
  16. Importing a wild card domain SSL certificate into AWS Certificate Manager
    10,348
  17. AWS Lambda Example based on Teraform
    900
  18. Simple AWS ECS example based on Teraform
    988
  19. Web front-end architecture in AWS for e-commerce website
    365
  20. Web Application Hosting provides the reliable, scalable, secure, and high-performance infrastructure
    10,615
  21. Time series processing architecture in AWS
    10,559
  22. Synchronous online gaming architecutre in AWS - Highly Available, Scalable & Elastic to support Millions of Players
    10,417
  23. Secure and Scalable LAMP Service Architecture in AWS
    10,660
  24. Live streaming / Static Content & Media Service based on AWS
    10,655
  25. Large scale messenger service architecture in AWS - Slack case study
    11,674
  26. Large scale computing based on huge data sets in AWS
    10,609
  27. Image Moderation Chatbot - a serverless chatbot on AWS that monitors chat channels and removes images suggestive or explicit image
    10,426
  28. Financial services grid computing in AWS
    10,354
  29. File synchronization service architecture in AWS
    10,376
  30. Fault Tolerance & High Availability based on multi Availability zones in AWS
    10,805
Recently updated
  1. Why 100% SEO Guarantees Better User Engagement and How to Achieve It in Your Service
    40
  2. Why SPF and DKIM Are Essential for Linux Email Servers?
    34
  3. How do I create animated background for Google Meet?
    20,613
  4. MaxMind GeoLite2 vs. DB-IP Lite vs. ip-api.com: Choosing the Right Free IP Geolocation Solution
    244
  5. Meet Valkey: Why Redis Changed Its License, Why Valkey is Better, and How to Use It as a Drop-in Replacement
    94
  6. How Rogue Web Scrapers Are Stealing My Content and Destroying My Server
    157
  7. Array of arguments passed to script in PHP
    90
  8. Invalid Auction Storms in Ad Tech
    98
  9. Building MRC-Compliant Instrumentation in Ad Tech: Implementation Steps and Key Challenges
    202
  10. The Two Clocks of Ad Server Architecture
    70
  11. Frequency Cap Architecture Overview
    514
  12. Collecting Proof-of-Play from 100,000 Screens
    146
  13. Vector Search in Relational Databases (RDBs): Trends & Usage Environments
    287
  14. What is RedisVL?
    109
  15. Designing a Skill Registry Architecture: Minimize Token Usage and Maximize AI Performance
    230
  16. TPU & GPU Cluster Management Tools
    244
  17. The Concept of Scaffolding in AI Agents
    233
  18. Beyond Static Pipelines: Architecting LLM Multi-Agent Systems for Autonomous Workflow Orchestration
    583
  19. AI-DLC vs. AI-SDLC: Model-Centric vs. System-Centric AI Development
    258
  20. Fine-Tuning Open-Weight LLMs: Training, Licensing, and Deploying Custom Models with Ollama
    339
  21. Inside the Next-Generation Streaming Architecture: Ultra-Low Latency, SGAI, and AI Codecs
    202
  22. Machine Learning: Training vs. Ranking
    221
  23. The Continuous Learning Machine: Designing a Loop That Actually Turns
    222
  24. The Next-Gen Low-Latency Standards: LL-HLS & MoQ
    195
  25. Designing a Data Curation System Around a Vector Database
    220
  26. Extracting Context from Live Video: The Architecture Got Simpler When We Stopped Building Features
    256
  27. The New Ad Stack Behind Live Sports Streaming: Inside SGAI, EABN, and the Standards Race to Fix Live Monetization
    656
  28. Designing MLOps Architectures to Handle Seasonal Data Drift in EV Fleet Models
    195
  29. From 15 to 40+ FPS: Accelerating PyTorch Object Detection on Automotive ECUs
    217
  30. Understanding Data Lineage
    260