Issue 2022-08-18: Surprise Cloud Bills

How to prevent them and what to do when you get one

Author

Keith Alpichi
August 18, 2022

One of the most common oopsies in using the Cloud is inadvertently leaving paid resources running and racking up a large bill.

This Reddit user racked up a $27k bill because they forgot to ensure orphaned EBS volumes were terminated.

This Reddit user racked up a $118 bill because they forgot to terminate EC2 instances.

These stories seem like simple preventable mistakes. The solution seems obvious- don't be forgetful. Sure, they're preventable but due to how complex these cloud providers are it can be easy to miss something. The less experience you have the higher the risks of making these mistakes.

Misconfiguring the wrong thing or overlooking something could cost you, literally. When you do get that surprise bill what do you do?

The Learnings

Let's talk about how you can prevent surprise charges.

Understand the Service

Here is a more realistic solution than not being forgetful- fully understand the service you're using. That includes understanding how every thing you select and configure relates to cost. If you don't know what it means, stop what you're working on and consult the documentation.

Setup Budgets

Every cloud provider has some features that allow you to set budgets and get alerts when you exceed them. This is a good safety measure to ensure you can take action as soon as possible. If those two Reddit users had alerts they could have caught the issue much sooner.

Use Infrastructure-as-Code (IaC)

Leverage Infrastructure-as-Code instead of interactively provisioning resources in the browser. This when you define in a text file the cloud resources you want built and destroyed. You provide this file to the cloud provider and they build the resources for you in an automated reproducible manner. When you're done running resources and you want them destroyed it's as simple as instructing the cloud provider to terminate all them. Using IaC makes it easy to not miss anything.

Some popular tools include AWS Cloudformation, Hashicorp Terraform, AWS CDK, and Pulumi. What do I recommend? I'm a big fan of Terraform.

Get Familiar with the Billing Dashboards

All cloud providers have a service to query, filter, and breakdown current and predicted costs. For AWS, the Billing and Cost Managment service gives you:

  • access to your current and past bills

  • insight into services that are accruing high costs

  • insight into cost trends

  • the ability to run reports to gain an understanding of costs against services, regions, accounts, instance types, tags, and more

Now let's talk about what you can do when you get a surprise bill.

Reach Out to Support

These mistakes happen so often that cloud providers are often empathetic and forgiving. Say you get a surprise large bill. Immediately submit a support ticket. Be calm, humble and honest when explaining your situation. Be descriptive in your ticket description. Be responsive and respectful to your assigned support engineer. If your situation is deemed forgivable you may get a fair amount of your money back.

Stop the Charges from Increasing

Leverage the billing dashboards to understand the culprit of the charges then take action. That action may be to terminate, disable, or reconfigure the resource or service. Lower your budget alert thresholds and review the service documentation to prevent the mistake from happening again.

Recent Oopsies