CloudWatch Metrics Data Gap Detected – Fix Guide

You're missing data points in CloudWatch metrics. This usually means an instance stopped reporting, a custom metric failed, or there's a threshold breach. Let's fix it.

You're staring at a CloudWatch graph and there it is—a gap. Maybe a flat line where data should be. Or your alarm fires with "Insufficient data" because the metric just stopped reporting. This happens after you deploy a new EC2 instance and forget to install the CloudWatch agent. Or you set up a custom metric from a Lambda function, but the function threw an unhandled exception and never sent the data. I've also seen it when an Auto Scaling group terminates an instance mid-reporting, leaving a 5-minute hole.

What Actually Causes the Gap

Three things cause this, and they're all straightforward:

The data source stopped sending. Could be the CloudWatch agent crashed, the EC2 instance went down, or your custom app just stopped emitting metrics. The agent on Linux, for example, runs as a systemd service. If that service stops, the data stops.
The metric namespace changed. You might have switched from AWS/EC2 to a custom namespace like MyApp/CPU but kept the alarm pointing at the old one. CloudWatch doesn't warn you—it just shows a gap.
You passed the retention period. Metrics in CloudWatch have a retention policy. For custom metrics, it's 15 months. But if you're looking at a dashboard and the time range exceeds the metric's age, you'll see a gap. That's not a failure, it's just old data being cleaned up.

The real fix depends on which of these three you're dealing with. Let's walk through each scenario.

How to Fix a CloudWatch Metrics Data Gap

Start by confirming the metric namespace and dimension. Go to the CloudWatch console, select Metrics, and search for your metric. If you don't see it at all, the source isn't sending. If you see older data but a gap recently, the source stopped recently.

Check the CloudWatch agent status on the source instance. SSH into the EC2 instance and run:
```
sudo systemctl status amazon-cloudwatch-agent
```
If it shows as inactive or dead, restart it with:
```
sudo systemctl restart amazon-cloudwatch-agent
```
After restarting, wait 2 minutes and check the CloudWatch console again. You should see new data points appear.
If the agent is running, check its logs. Look at /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log. Search for errors like "permission denied" or "failed to collect". A common one: the agent can't read the disk metrics because the disk plugin isn't enabled. Edit the agent config file at /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json and add the disk plugin if it's missing. Then restart the agent.
For custom metrics from Lambda or your app. Check the application logs for errors when it tries to publish to CloudWatch. If you're using the AWS SDK, look for PutMetricData calls. A common mistake: the IAM role doesn't have cloudwatch:PutMetricData permission. Verify the role attached to the Lambda function or EC2 instance has that policy. You can test it by running a quick AWS CLI command from the instance:
```
aws cloudwatch put-metric-data --metric-name TestMetric --namespace TestNamespace --value 1 --unit Count
```
If that fails, you get an access denied error. Fix the IAM policy, then retry.
Check if the metric retention period expired. This only matters if you're looking at old data. Go to the Metrics tab in CloudWatch, select your metric, and look at the time range. If you set the dashboard to show the last 3 months but the metric was only created 2 months ago, you'll see a gap at the beginning. That's expected. Adjust the time range to match the metric's lifetime.
If the gap is in a dashboard or alarm. Verify the alarm's metric query. Go to the alarm, click Edit, and look at the Metric section. Make sure the namespace, metric name, and dimensions exactly match what's being published. I once spent an hour debugging a gap only to find I was looking at AWS/EC2 but the instance was publishing under Custom/EC2. Fix the alarm to use the right namespace.

What to Check If the Gap Persists

If you've done all five steps and data is still missing, here's the checklist:

Network connectivity. The CloudWatch agent needs outbound HTTPS access to monitoring.us-east-1.amazonaws.com (or your region's endpoint). If the instance is in a private subnet with a NAT gateway, make sure the NAT gateway is healthy. You can test connectivity with curl -v https://monitoring.us-east-1.amazonaws.com.
Agent configuration file syntax. The JSON file can be tricky. A missing comma or an extra bracket breaks the whole config. Use a JSON validator like jq: cat amazon-cloudwatch-agent.json | jq. If it errors out, the config is bad.
Multiple agents running. If you have both the old CloudWatch monitoring scripts and the new unified agent, they conflict. Remove the old scripts: sudo yum remove aws-cfn-bootstrap on Amazon Linux, or sudo apt-get remove aws-cfn-bootstrap on Ubuntu.
Check the region. You might be publishing to us-west-2 but your dashboard is showing us-east-1. CloudWatch is region-specific. Verify you're looking at the same region where the data is being sent.

One last thing: don't waste time on the CloudWatch console's built-in troubleshooting. It's vague and rarely points to the real issue. Stick to logs and CLI commands.

Real-world example: A client had a gap every day at 3 AM. Turned out the nightly backup script was stopping the CloudWatch agent before running, and forgetting to restart it. After adding the restart command to the backup script the gap disappeared.