Not Getting AWS Data? You May Need To Increase Your AWS Cloudwatch API Limit

If you are using the CopperEgg AWS monitoring (thanks for doing so!) and are a heavy AWS user with many monitored objects, you may run across an issue where data is not appearing for some of your objects.  Here's an example of what this looks like.

awsissue.png

 

Why does this happen?  Well, Amazon has a lot of limits they put on customer accounts to prevent runaway usage.  Pretty much every kind of asset they have in their system has a default limit on it. You have probably already run across the EC2 instance limits, EBS number and total size limits, and so on.  Well, there is another limit - the number of requests you can make to AWS APIs.  Since we are making requests on behalf of your account, both our monitoring traffic and any other hits you or another tool you're using make to the Cloudwatch APIs all count against this limit.  See the AWS CloudWatch Developer Guide for information on these limits. The default limit is 1M hits/month - which sounds like a lot, but if you are hitting the API every 5 minutes to get information for each object (EC2, EBS, Elasticache, DynamoDB, RDS) you have, that's only 3472 total objects, and that's only if you never hit the API for any other reason. It's pretty easy to hit the limit even with just a couple hundred servers (especially if they use EBSes).

We do retries in our code to get around short-term throttling, but if you're going over your general API cap the real fix is to request an increase from AWS.  Like many of the limits, it's not really shown to you in the console or API but they do let you put in support requests to up your limit.  Your AWS representative and AWS support should both be able to help you. If they ask, just explain that you are using a CloudWatch monitoring tool that makes a lot of ListMetrics andGetMetricStatistics calls. You could probably start with doubling your limit, though if you have significant environment growth planned I'd factor that in.  Their API limits don't resize with how many AWS objects you have, so this is a pretty routine operation for them; any serious user runs across it sooner or later and needs an increase.  Contact support-uptimecm@idera.com if you have any issues.

 

 

 

 

Powered by Zendesk