Your Reserved Instance metrics could be deceiving youErik Carlin, Co-Founder and Chief Product Officer
- Commonly used metrics like RI utilization and RI coverage don't always correlate with savings. Lower RI utilization and RI coverage could generate higher savings.
- Effective Savings Rate (ESR) is a simple, single metric that always reflects your actual RI savings rate. It should be the one metric you seek to maximize and use to measure RI savings performance.
- If you don't know your ESR, you're probably saving less than you think. ProsperOps offers a free, no commitment RI Savings Analysis that shows your historical ESR trend, as well as future savings potential with an RI portfolio actively managed by ProsperOps.
This is part 2 of a 3 part series introducing the concept of Effective Savings Rate (ESR). In part 1, we describe what ESR is, why it's necessary, and how to calculate it. In this post, we'll look at how existing Reserved Instance (RI) metrics can be misleading and why ESR is the one metric that matters when measuring RI savings performance. In part 3 of the series, we'll show you how to calculate your own ESR and discuss good and bad ESRs.
"My RI utilization is 95% so we're good." It's not uncommon for us to hear this and our response is usually the same: "Would you rather have high RI utilization or high RI savings?" (you could also substitute coverage for utilization) People generally pause at that question because they equate high RI utilization or coverage with high RI savings. Everyone wants high savings, but they measure it via utilization and coverage. The problem with that approach is that both RI utilization and RI coverage can be misleading when it comes to actual dollars saved with RIs. If your goal is to maximize savings, you need a metric that always correlates with savings. This is Effective Savings Rate (ESR).
The skeptic in you may be thinking, "If ESR is so critical, why hasn't someone introduced it previously? RI utilization and RI coverage seem to be what AWS and every RI reporting tool highlight, so aren't we good?"
The answer is "no!" Keep reading and we'll prove it to you (we're skeptics by nature too). We're not sure why ESR hasn't been introduced until now. Regardless, as a fully automated RI management service we needed a definitive way to quantify the savings outcomes of every possible RI portfolio combination. ESR was born from that need and we've learned over time just how unreliable other RI metrics can be. Let's take a look at why ESR is really the one metric that matters.
Quick recap: ESR represents the percentage discount you receive with RIs across your entire EC2 fleet, when all is said and done. With EC2 instances coming and going, RIs being bought, modified, exchanged, and expiring, each with differing discount rates, across multiple AWS accounts, it's non-trivial to understand your actual savings rate. That's what ESR does—it's the ultimate RI savings truth teller (read part 1 of this series for more background on ESR).
Limitations of RI coverage
RI coverage is a common RI metric and represents the percent of EC2 "covered" with RIs. For example, if you have 10 EC2 instances and five corresponding RIs, your RI coverage is 50%. If four RIs match but one RI doesn't, your RI coverage is 40%. And so forth.
Since RIs offer a discount, the higher the RI coverage, the more money you'd expect to save, right? If your goal is to maximize savings (which is your ultimate goal), does higher RI coverage mean higher savings? (hint: it's a trick question)
Consider the following 2 environments, each consisting of the same type and number of EC2 instances:
In Environment A, we cover three of our four t3.small instances with RIs and have a coverage of 75%. Not too shabby. With that setup, our monthly cost is $47.38.
In Environment B, we cover all four of our t3.small instances with RIs so our RI coverage is 100%. Nice! But, we also have two extra t3.small RI commitments which do not have corresponding instances. The monthly cost of this environment is higher at $64.80.
Both environments are running four t3.small instances but the environment with lower coverage has the higher savings. This means RI coverage alone can not be trusted to maximize savings.
Something else that might not be obvious is that even with 100% RI coverage in Environment B, we paid more by using RIs than if we just ran everything on-demand at a monthly cost of $55.90. Even with perfect RI coverage, we lost money!
Now, let's look at ESR for the two environments. We see that in Environment A we saved 20.9%, while in Environment B we had a savings rate of -8.2%. Unlike RI coverage, ESR accurately reflects the savings dynamic at play. Although RI coverage doesn't correlate with savings, ESR always does.
Bottom line: Seek to maximize ESR not RI coverage.
Limitations of RI utilization
The second common RI metric is RI utilization. This represents what percent of your RI portfolio is being "used" and matches up with corresponding EC2 instances. For example, if you have 5 RIs that match 4 EC2 instances, your RI utilization is 80%. If that same RI portfolio only has a single matching EC2 instance, the RI utilization is 20%.
As with RI coverage, you would expect that higher RI utilization rates equal higher savings. By now you've probably realized that's not always the case.
Consider the following 2 environments, each consisting of the same type and number of EC2 instances:
In Environment A, we cover all five of our t3.small instances with RIs but have a single unutilized t3.small RI. Since five of the six RIs have corresponding instances, the resultant RI utilization is 83%. Not too bad. The monthly cost of the environment is $64.80.
In Environment B, which has the same number of t3.small instances, we have a single t3.small RI. Because the RI is matched, the RI utilization is 100%, but the monthly cost is higher at $70.70.
Running five t3.small instances 100% on-demand for a month would cost $74.88. Both the Environment A and Environment B RI portfolios saved us money, but the environment with the lower RI utilization had the higher savings. Again, just as with RI coverage, RI utilization alone can not be trusted to maximize savings.
Let's look at ESR. Does it help reveal the actual savings dynamic at play here? We see that ESR is 13.5% for Environment A and 5.6% for Environment B. So yes, unlike RI utilization, ESR accurately reflects savings performance. Although RI utilization doesn't correlate with savings, ESR always does.
Bottom line: Seek to maximize ESR not RI utilization.
Limitations of RI coverage and RI utilization together
Astute readers and RI practitioners might say, "You need to maximize for both high RI coverage and high RI utilization together." OK, but how do you best do that? For example, would you rather have an environment with 80% utilization and 90% coverage or 90% utilization and 80% coverage? It's unclear which delivers superior savings. Why deal with two metrics when you can simplify down to one metric that always correlates with savings? Simply put, maximizing ESR will always result in maximum savings for you.
There are also other ways of increasing savings that RI utilization and RI coverage, even when looked at together, simply don't account for. For example, consider the following 2 environments, each consisting of the same type and number of EC2 instances:
In Environment A, we cover our Linux t2.2xlarge instance with an RI, while in Environment B, we cover the Windows m5.xlarge instance instead. The RI coverage and RI utilization for both environments is identical, but Environment A has a monthly cost of $461.16 while Environment B is more expensive at $501.26 per month. Even with same RI coverage and RI utilization, savings can still vary!
Again, ESR is our guide and truth teller for savings. The ESR for Environment B is 6.8%, while Environment A is 14.3%, more than twice as much.
It's important to note here that in either environment, we are making nearly identical dollar commits to AWS. In Environment A, we're covering a Linux t2.2xlarge instance with an annual on-demand cost of $3,251.71 and in Environment B we're covering a Windows m5.xlarge instance with an annual on-demand cost of $3,293.76. Using a convertible RI, we're making a dollar commitment to AWS for roughly the same amount of money, but all things being equal, we're much better off covering the Linux instance due to the higher RI discount rate. A combination of RI coverage and RI utilization doesn't give us the data we need to maximize savings, but ESR does.
Bottom line: Seek to maximize ESR not RI coverage and RI utilization together.
The above are just simple examples to illustrate the pitfalls of relying on RI utilization and RI coverage. The real world is more complex, but the fundamental issues with these metrics remain and are often magnified in practice.
Putting it all together in practice
We talk to lots of customers about RIs. When we ask about the effectiveness of current strategies and tooling to deliver RI savings, we generally see several things:
- Effectiveness is almost always spoken of in terms of RI coverage and/or RI utilization, which as we've seen above, doesn't necessarily correlate with actual savings.
- Effective Savings Rates aren't known (we're trying to help with that).
- Guesstimates of ESR are almost always higher than what the actual ESR is, once measured. Managing RIs is hard and complex work!
It's not uncommon for us to see a historical savings pattern that looks something like this:
RIs were being used to generate a modest ESR of ~7% (we'll talk about good and bad ESRs in post 3) but RIs expired, the environment became more complex and elastic, etc. The ESR eventually turns negative, which means the cost of using RIs now exceeds the cost of running all instances on-demand.
Do you know your ESR? If not, you may be saving less than you think, or worse, losing money. Our mission at ProsperOps is to simplify the RI savings experience (save you time), automate the complex RI work required to maximize savings (save you money), and ultimately be a part of helping businesses prosper in the cloud.
We offer a free, no commitment RI Savings Analysis that shows your historical ESR trend as well as future savings potential with an RI portfolio actively managed by ProsperOps.
To learn more or sign-up for a free, no commitment RI Savings Analysis, please visit https://www.prosperops.com.