A Costly Mistake: How an Empty S3 Bucket Led to a Massive AWS Bill

AWS Customer Faces Massive Bill Due to Open-Source Tool Misconfiguration.

In a startling incident, an AWS customer faced a staggering $1,300 bill for S3 usage, despite creating a single, empty bucket for testing purposes.

The culprit? A popular open-source tool with a default configuration that stored backups in the customer’s S3 bucket, leading to a deluge of unauthorized requests.

The customer, who remained anonymous, shared their experience in a detailed blog post, shedding light on the unexpected costs and potential security risks associated with misconfigured tools and S3 bucket naming conventions.

The Unexpected Bill

After creating a private S3 bucket in the eu-west-1 region for a proof-of-concept document indexing system, the customer was shocked to discover nearly 100,000,000 S3 PUT requests executed within a single day, resulting in a bill exceeding $1,300.

Upon investigation, the customer discovered that a popular open-source tool had a default configuration that used the same bucket name as their private S3 bucket for storing backups.

As a result, every deployment of this tool with the default configuration attempted to store its backups in the customer’s bucket, leading to a massive influx of unauthorized requests.

In a response from AWS support, the customer learned that S3 charges for unauthorized requests (4xx errors) as well, even if the requests are denied.

This meant that the customer was responsible for paying for the millions of unauthorized requests made to their bucket.

In a concerning experiment, the customer briefly made their bucket public for writes, collecting over 10GB of data within 30 seconds.

“One of the popular open-source tools had a default configuration to store their backups in S3. And, as a placeholder for a bucket name, they used… the same name that I used for my bucket. This meant that every deployment of this tool with default configuration values attempted to store its backups in my S3 bucket!” user informed via blog post.

This highlighted the potential for data leaks and security breaches due to misconfigured systems attempting to write data to unintended S3 buckets.

Combat Sophisticated Email Threats With AI-Powered Email Security Tool ->  Try Free Demo 

Lessons Learned

The incident underscored several important lessons:

1. Bucket Naming Conventions: Adding random suffixes to S3 bucket names can enhance security by reducing vulnerability to misconfigured systems or intentional attacks.
2. Specifying AWS Regions: When executing numerous requests to S3, explicitly specifying the AWS region can avoid additional costs from S3 API redirects.
3. Unauthorized Request Charges: AWS charges for unauthorized requests, even if denied, which can lead to unexpected costs if not properly monitored.

Recommendations

The customer reported the issue to the maintainers of the vulnerable open-source tool, who promptly fixed the default configuration. However, existing deployments may still be affected.

AWS was notified, but they were unwilling to address misconfigurations of third-party products. The customer also attempted to inform companies whose data was in their bucket, but received no response.

Ultimately, AWS cancelled the customer’s S3 bill as an exception but emphasized that such exceptions are not guaranteed.

This incident is a cautionary tale for AWS customers to carefully monitor their S3 usage, implement secure bucket naming conventions, and be aware of the potential costs and security risks associated with misconfigured tools and unauthorized requests.

Is Your Network Under Attack? - Read CISO’s Guide to Avoiding the Next Breach - Download Free Guide