AWS S3 IAM errors with missing files: 404 expected, 403 returned

“Why is AWS S3 returning authorisation errors (HTTP 403 responses) for bucket files that don’t exist (rather than HTTP 404 responses)? The IAM role has the correct permissions (s3:GetObject). If the bucket file is available - the content is returned as normal. This makes no sense, what is going on…” 🙇‍♂️

This was me a few weeks ago. After losing too many hours debugging this issue (with no success), I reached out to Ant Stanley to check what I was missing. He started laughing and pointed me to this tweet. I’d run into the exact same issue…

As stated in the AWS documentation, detecting missing bucket files relies on more than the s3:GetObject permission.

If the object you request does not exist, the error Amazon S3 returns depends on whether you also have the s3:ListBucket permission.

• If you have the s3:ListBucket permission on the bucket, Amazon S3 will return an HTTP status code 404 (“no such key”) error.
• If you don’t have the s3:ListBucket permission, Amazon S3 will return an HTTP status code 403 (“access denied”) error.

As I’d been following the security principle of least privilege for applications - I’d only added the minimum set of permissions to interact with bucket files (s3:GetObject & s3:PutObject). This was configured with the following CloudFormation IAM policy.

{
  "PolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [      
      {
        "Effect": "Allow",
        "Action": [
          "s3:PutObject",
          "s3:GetObject"
        ],
        "Resource": [
          "arn:aws:s3:::bucket-id/*",
        ]
      }
    ]
  }
}

This worked fine until I needed to check for missing files and encountered the unexpected error response. Assuming I’d misconfigured IAM in some more fundamental way, I had a frustrating time failing to debug this - until Ant pointed me in the right direction.

Here is the CloudFormation IAM policy configuration I needed to use. It added the s3:ListBucket permission for the bucket resource.

{
  "PolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "s3:ListBucket"
        ],
        "Resource": [
          "arn:aws:s3:::bucket-id"
        ]
      },
      {
        "Effect": "Allow",
        "Action": [
          "s3:PutObject",
          "s3:GetObject"
        ],
        "Resource": [
          "arn:aws:s3:::bucket-id/*",
        ]
      }
    ]
  }
}

I’m sure I won’t be the last person to encounter this confusing “feature” of AWS S3. Hopefully this blog post will turn up in some future developer’s search results when they are trying to diagnose this issue (and save them a few hours of frustrated debugging). Thanks to Ant Stanley for saving me losing any more time on this issue 👏👏👏.