Amazon Macie: Automated Sensitive Data Discovery and S3 Security Posture

S3 buckets accumulate data in ways nobody fully tracks. A developer uploads a CSV for testing that turns out to contain real customer email addresses. An application logs request bodies that include credit card numbers during debugging. An ETL job writes a file to the wrong bucket. Without automated scanning, these situations persist until an audit or a breach surfaces them.

Amazon Macie uses machine learning to continuously scan S3 buckets for sensitive data and security misconfigurations. It identifies PII categories like names, addresses, and government IDs, financial data like credit card and bank account numbers, and credentials like API keys and passwords. It also surfaces security findings like publicly accessible buckets, unencrypted objects, and buckets that have had their access control modified. In this article I will walk through enabling Macie across an organization with Terraform, creating custom data identifiers, and automating remediation.

Enabling Macie at the Organization Level

resource "aws_macie2_account" "main" {
  finding_publishing_frequency = "FIFTEEN_MINUTES"
  status                       = "ENABLED"
}

resource "aws_macie2_organization_admin_account" "main" {
  admin_account_id = var.security_account_id

  depends_on = [aws_macie2_account.main]
}

resource "aws_macie2_member" "accounts" {
  for_each   = toset(var.member_account_ids)
  account_id = each.value
  email      = var.member_account_emails[each.value]
  invite     = true

  depends_on = [aws_macie2_organization_admin_account.main]
}

Enabling Macie through the organization ensures every member account’s S3 buckets are scanned from the security account without requiring individual account configuration. The delegated administrator account receives all findings from all member accounts in a single findings feed. This is the correct architecture for organizations with multiple accounts, which is most production environments using AWS Organizations.

Classification Jobs

resource "aws_macie2_classification_job" "pii_scan" {
  name       = "pii-discovery-scan"
  job_type   = "SCHEDULED"

  schedule_frequency {
    weekly_schedule = "MONDAY"
  }

  s3_job_definition {
    bucket_definitions {
      account_id = data.aws_caller_identity.current.account_id
      buckets    = [
        aws_s3_bucket.data_lake.id,
        aws_s3_bucket.application_logs.id,
        aws_s3_bucket.user_uploads.id
      ]
    }

    scoping {
      excludes {
        and {
          simple_scope_term {
            comparator = "STARTS_WITH"
            key        = "OBJECT_KEY"
            values     = ["processed/", "archived/"]
          }
        }
      }
    }
  }

  sampling_percentage = 100

  tags = {
    Environment = "production"
    Purpose     = "compliance-scanning"
  }
}

sampling_percentage = 100 means Macie scans every object in the specified buckets rather than a sample. For compliance purposes where you need to certify that sensitive data is not present in a bucket, full scanning is required. For large buckets where full scanning would be cost-prohibitive, use sampling and accept that findings represent a statistical assessment rather than a complete inventory.

Custom Data Identifiers

resource "aws_macie2_custom_data_identifier" "employee_id" {
  name        = "employee-id-pattern"
  description = "Matches internal employee ID format: EMP followed by 6 digits"
  regex       = "EMP[0-9]{6}"

  keywords             = ["employee", "staff", "emp_id"]
  maximum_match_distance = 50
  ignore_words         = ["TEMPLATE", "EXAMPLE", "SAMPLE"]
}

resource "aws_macie2_custom_data_identifier" "api_key_pattern" {
  name        = "internal-api-key"
  description = "Matches internal API key format"
  regex       = "sk_[a-zA-Z0-9]{32,}"

  keywords           = ["api_key", "secret_key", "authorization"]
  maximum_match_distance = 100
}

Custom data identifiers extend Macie’s built-in detection to cover your organization’s specific sensitive data formats. Internal employee IDs, proprietary document reference numbers, API key patterns specific to your platform, and internal classification codes can all be detected using regular expressions with keyword context requirements. The keywords requirement means the regex must appear within maximum_match_distance characters of one of the keywords, which reduces false positives significantly.

Automated Remediation via EventBridge

resource "aws_cloudwatch_event_rule" "macie_findings" {
  name        = "macie-high-severity-findings"
  description = "Route high severity Macie findings to remediation Lambda"

  event_pattern = jsonencode({
    source      = ["aws.macie"]
    detail-type = ["Macie Finding"]
    detail = {
      severity = {
        description = ["High", "Critical"]
      }
    }
  })
}

resource "aws_cloudwatch_event_target" "remediation" {
  rule = aws_cloudwatch_event_rule.macie_findings.name
  arn  = aws_lambda_function.macie_remediation.arn
}
import boto3
import json
import logging

logger = logging.getLogger()
s3     = boto3.client("s3")

def lambda_handler(event, context):
    finding      = event["detail"]
    finding_type = finding["type"]
    severity     = finding["severity"]["description"]
    resources    = finding["resourcesAffected"]

    if "Policy:IAMUser/S3BucketPublic" in finding_type or "Policy:IAMUser/S3BucketSharedExternally" in finding_type:
        bucket_name = resources["s3Bucket"]["name"]
        remediate_public_bucket(bucket_name)

    logger.info(json.dumps({
        "action":       "macie_finding_processed",
        "finding_type": finding_type,
        "severity":     severity,
        "resources":    resources
    }))

def remediate_public_bucket(bucket_name: str):
    logger.warning(f"Blocking public access on bucket: {bucket_name}")
    s3.put_public_access_block(
        Bucket = bucket_name,
        PublicAccessBlockConfiguration = {
            "BlockPublicAcls":       True,
            "IgnorePublicAcls":      True,
            "BlockPublicPolicy":     True,
            "RestrictPublicBuckets": True
        }
    )

Automated remediation for public bucket findings is safe to implement because a bucket being made public is almost never intentional in a production environment and the remediation is non-destructive. The data remains intact, the policy is simply corrected. For findings about sensitive data content, automated remediation is riskier and typically involves alerting the data owner rather than taking action automatically.

Closing Thoughts

Macie is the automated answer to a question that is otherwise answered by manual audit: do any of our S3 buckets contain data they should not contain, and are they configured correctly to protect the data they should contain? Without it, the answer is effectively unknown between audits. With it, the answer is updated continuously.

Enable it at the organization level from your security account. Run classification jobs on a weekly schedule against your most sensitive buckets at 100 percent sampling. Add custom data identifiers for your organization-specific sensitive data formats. Automate remediation for the findings where the correct action is unambiguous. The operational cost is minimal and the compliance confidence is significant.

Enjoy the cloud.

Osama


#AWS #AmazonMacie #DataSecurity #CloudSecurity #CloudGovernance #Terraform #InfrastructureAsCode #AmazonWebServices #SolutionsArchitect #CloudComputing #DataPrivacy #PII #Compliance #TechBlog #CloudInfrastructure #S3Security #GDPR #DataProtection #SecurityAutomation #DevSecOps

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.