An S3 data lake without governance is a compliance liability and an operational headache. IAM alone cannot give you column-level security on Parquet files. There is no central place to see who has access to what. Data discovery depends on asking colleagues rather than running queries. Sensitive columns like SSN and credit card numbers need the same access controls as a database, not a storage bucket.

AWS Lake Formation adds a governance layer on top of S3 and the Glue Data Catalog. It provides a central permissions model that works across Athena, Redshift Spectrum, Glue, and EMR, column-level and row-level security on catalog tables, tag-based access control at scale, and data lineage tracking. In this article I will walk through setting it up with Terraform and implementing fine-grained access control.

Enabling Lake Formation and Registering S3 Locations

resource "aws_lakeformation_data_lake_settings" "main" {
  admins = [
    aws_iam_role.lake_formation_admin.arn,
    data.aws_caller_identity.current.arn
  ]

  create_database_default_permissions {
    permissions = []
    principal   = "IAM_ALLOWED_PRINCIPALS"
  }

  create_table_default_permissions {
    permissions = []
    principal   = "IAM_ALLOWED_PRINCIPALS"
  }
}

resource "aws_lakeformation_resource" "data_lake" {
  arn      = aws_s3_bucket.data_lake.arn
  role_arn = aws_iam_role.lake_formation_s3.arn
}

resource "aws_lakeformation_resource" "curated" {
  arn      = "${aws_s3_bucket.data_lake.arn}/curated/"
  role_arn = aws_iam_role.lake_formation_s3.arn
}

Setting create_database_default_permissions and create_table_default_permissions to empty arrays for IAM_ALLOWED_PRINCIPALS is the critical first step. By default, Lake Formation grants all permissions to IAM_ALLOWED_PRINCIPALS, which means existing IAM policies still control access and Lake Formation does nothing. Clearing these defaults forces all access to go through Lake Formation permissions. Do not skip this step or your governance layer has no effect.

LF-Tags for Attribute-Based Access Control

resource "aws_lakeformation_lf_tag" "sensitivity" {
  key    = "Sensitivity"
  values = ["public", "internal", "confidential", "restricted"]
}

resource "aws_lakeformation_lf_tag" "domain" {
  key    = "Domain"
  values = ["orders", "customers", "products", "finance", "hr"]
}

resource "aws_lakeformation_resource_lf_tags" "orders_table" {
  table {
    database_name = "curated_data"
    name          = "orders"
  }

  lf_tag {
    key   = "Sensitivity"
    value = "internal"
  }

  lf_tag {
    key   = "Domain"
    value = "orders"
  }
}

resource "aws_lakeformation_resource_lf_tags" "customers_pii_columns" {
  table_with_columns {
    database_name = "curated_data"
    name          = "customers"
    column_names  = ["email", "phone", "address", "date_of_birth"]
  }

  lf_tag {
    key   = "Sensitivity"
    value = "restricted"
  }
}

resource "aws_lakeformation_tag_based_permissions" "analyst_orders" {
  principal     = aws_iam_role.data_analyst.arn
  permissions   = ["SELECT"]
  catalog_id    = data.aws_caller_identity.current.account_id

  lf_tag_policy {
    resource_type = "TABLE"

    expression {
      key    = "Sensitivity"
      values = ["public", "internal"]
    }

    expression {
      key    = "Domain"
      values = ["orders", "products"]
    }
  }
}

resource "aws_lakeformation_tag_based_permissions" "pii_team" {
  principal   = aws_iam_role.pii_analyst.arn
  permissions = ["SELECT"]
  catalog_id  = data.aws_caller_identity.current.account_id

  lf_tag_policy {
    resource_type = "COLUMN"

    expression {
      key    = "Sensitivity"
      values = ["public", "internal", "confidential", "restricted"]
    }
  }
}

LF-Tags let you grant access by attribute rather than by specific table name. The data analyst role gets SELECT on any table tagged Sensitivity=public or internal in the orders or products domains. As new tables are created and tagged, they automatically fall into or outside this permission without any permission updates. This scales to hundreds of tables and dozens of teams without permission sprawl.

Row-Level Security

resource "aws_lakeformation_data_cells_filter" "emea_orders" {
  table_data {
    database_name    = "curated_data"
    name             = "orders"
    table_catalog_id = data.aws_caller_identity.current.account_id

    filter_name = "emea-region-filter"

    row_filter {
      filter_expression = "region = 'EMEA'"
    }

    column_names = ["order_id", "customer_id", "order_date", "amount", "status", "region"]
  }
}

resource "aws_lakeformation_permissions" "emea_analyst_filtered" {
  principal   = aws_iam_role.emea_analyst.arn
  permissions = ["SELECT"]

  data_cells_filter {
    database_name = "curated_data"
    table_name    = "orders"
    name          = "emea-region-filter"
    table_catalog_id = data.aws_caller_identity.current.account_id
  }
}

Data cells filters combine row filtering and column projection. The EMEA analyst role can only see rows where region = EMEA and only the six columns specified. Any Athena query this role runs against the orders table automatically has these filters applied server-side before results are returned. The analyst cannot see other regions or excluded columns regardless of how they write the query.

Closing Thoughts

Lake Formation is the governance infrastructure that makes a data lake enterprise-ready. Without it you have S3 storage with Glue metadata, which is a starting point. With it you have a governed platform where access control is centralized, auditable, and scalable across hundreds of datasets and dozens of teams.

Clear the default IAM_ALLOWED_PRINCIPALS permissions immediately after enabling Lake Formation. Use LF-Tags for scalable attribute-based access control. Apply data cells filters for teams that should only see a subset of rows or columns. The governance layer pays back in audit confidence and compliance clarity at every data review thereafter.

Enjoy the cloud.

Osama

---

#AWS #LakeFormation #DataLake #DataGovernance #DataEngineering #CloudArchitecture #Terraform #InfrastructureAsCode #AmazonWebServices #SolutionsArchitect #CloudComputing #Athena #Glue #DataSecurity #TechBlog #CloudInfrastructure #ABAC #FineSecurity #Analytics #Compliance