CloudWatch Log Stream Missing: Fix EC2 Logs Not Showing

When This Happens

You install the unified CloudWatch agent on an EC2 instance — Amazon Linux 2023 or Ubuntu 22.04 — following the official AWS docs. You run sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json and restart the agent. You go to the CloudWatch console, open your log group, and… nothing. The stream name is there, but the Last Event Time column says —. No log events load. Or worse, the stream never appears at all.

Root Cause

What's actually happening here is one of two things, almost always. First: the EC2 instance doesn't have the right IAM role attached. The agent needs permissions to logs:PutLogEvents and logs:CreateLogStream on the specific log group. AWS's managed policy CloudWatchAgentServerPolicy works, but if you used a custom policy and missed logs:DescribeLogGroups, the agent can't even confirm the group exists — so it silently fails.

Second: the agent config file points to a log file path that doesn't exist, or the wildcard pattern doesn't match. For example, you specify "/var/log/nginx/*.log" but Nginx writes to /var/log/nginx/error.log and access.log — that works. But if you wrote "/var/log/nginx/*.txt"? No matches, no data. The agent doesn't yell at you; it just sends nothing.

The Fix: Step by Step

Check the IAM role first — skip the config until this is right.
Go to the EC2 console, select the instance, and view its IAM role under Security. Click the role name to open IAM. Verify it has either the managed policy CloudWatchAgentServerPolicy or a policy that includes these actions on the correct log group ARN:
```
logs:PutLogEvents
logs:CreateLogStream
logs:DescribeLogGroups
```
If the role is missing, attach the policy and wait 30 seconds. Then restart the agent: sudo systemctl restart amazon-cloudwatch-agent.
Verify the log file path in the agent config.
Open your config file (usually /opt/aws/amazon-cloudwatch-agent/bin/config.json) and look at the logs section. It looks like:
```
"logs": {
  "logs_collected": {
    "files": {
      "collect_list": [
        {
          "file_path": "/var/log/syslog",
          "log_group_name": "my-app-logs",
          "log_stream_name": "{instance_id}"
        }
      ]
    }
  }
}
```
Run sudo ls -la /var/log/syslog on the instance. If the file doesn't exist, change the path or check why the app isn't writing logs. If the file exists but is empty, the agent won't send anything — you need at least one line of content.
Restart the agent and check its own logs.
The agent writes a log to /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log. Run sudo tail -100 /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log. Look for lines with ERROR or WARN. Common ones:
- Unable to put log events — this means a permissions issue. Double-check step 1.
- No matching files — your file path pattern is wrong. Fix the path and re-fetch the config.
- ThrottlingException — you hit API rate limits. Wait a minute; it recovers.
Force a re-fetch of the config.
Sometimes the agent caches a bad config. Run:
```
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
```
Then immediately check the agent log again. The agent should say Config fetched successfully and show the number of log files it's monitoring.
Test with a dummy log entry.
If everything looks correct but still no data, force a log line: echo "test entry $(date)" | sudo tee -a /var/log/syslog. Wait 30 seconds. Refresh the CloudWatch console. If the stream now shows the test line, the issue is that your app wasn't writing logs. If it still shows nothing, the problem is either the role (double-check the policy JSON) or a network issue — the instance can't reach CloudWatch endpoints. Check sudo curl https://logs..amazonaws.com — if it fails, your security group or NACL is blocking outbound HTTPS (port 443).

If It Still Fails

Check the agent version. Run /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent --version. Versions below 1.300032.0 had a known bug where the agent would silently drop logs if the stream name included special characters. Upgrade to the latest from the AWS S3 bucket: sudo wget https://s3..amazonaws.com/amazoncloudwatch-agent-/amazon_linux/arm64/latest/amazon-cloudwatch-agent.rpm (adjust for your OS and arch).

Also verify you're looking at the right log group. I've spent 20 minutes debugging only to realize I was in the wrong AWS account. Check the log group name in both the config and the console — exact match, case-sensitive.

If you're using a custom log stream name like {instance_id}, the agent expands it to the EC2 instance ID. In CloudWatch, look for a stream named i-0abcd1234efgh5678. Sort the streams by last event time descending — it might be at the bottom because older streams push new ones down.

One last thing: on some older Amazon Linux 2 instances, the amazon-cloudwatch-agent service isn't enabled by default. Run sudo systemctl enable amazon-cloudwatch-agent and sudo systemctl start amazon-cloudwatch-agent manually.