Hand-rolled ETL scripts work until they do not. They break silently when source schemas change, accumulate fixes nobody documents, and grow into systems that one person understands and everyone else avoids. The cost of maintaining them compounds over time.
AWS Glue gives you a serverless ETL platform built around a central data catalog. Crawlers infer schemas from your data sources. The catalog stores and versions those schemas. Glue jobs transform data using PySpark or Python shell. Job bookmarking tracks which data has already been processed so reruns are safe. In this article I will walk through building a production Glue pipeline with Terraform and the patterns that make it reliable.
The Glue Data Catalog
The Glue Data Catalog is a managed Apache Hive Metastore. It stores database and table definitions, schema information, partition metadata, and connection details for your data sources. Athena, EMR, Redshift Spectrum, and Glue jobs all read from it, which means you define a schema once and every query engine uses it.
Crawlers populate the catalog automatically. You point a crawler at an S3 prefix, a JDBC database, or a DynamoDB table. The crawler samples the data, infers the schema, and writes a table definition to the catalog. Run it on a schedule to pick up new partitions and schema changes.
resource "aws_glue_catalog_database" "raw" {
name = "raw_data"
description = "Raw ingested data from application sources"
}
resource "aws_glue_catalog_database" "curated" {
name = "curated_data"
description = "Transformed and validated data ready for analysis"
}
resource "aws_glue_crawler" "orders_raw" {
name = "orders-raw-crawler"
role = aws_iam_role.glue_crawler.arn
database_name = aws_glue_catalog_database.raw.name
description = "Crawls raw order data from S3"
s3_target {
path = "s3://${aws_s3_bucket.data_lake.bucket}/raw/orders/"
exclusions = ["**.tmp", "**/_SUCCESS"]
sample_size = 100
}
schema_change_policy {
update_behavior = "UPDATE_IN_DATABASE"
delete_behavior = "LOG"
}
recrawl_policy {
recrawl_behavior = "CRAWL_NEW_FOLDERS_ONLY"
}
lineage_configuration {
crawler_lineage_settings = "ENABLE"
}
schedule = "cron(0 6 * * ? *)"
tags = {
Environment = "production"
ManagedBy = "terraform"
}
}
resource "aws_iam_role" "glue_crawler" {
name = "glue-crawler-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "glue.amazonaws.com" }
Action = "sts:AssumeRole"
}]
})
}
resource "aws_iam_role_policy_attachment" "glue_service" {
role = aws_iam_role.glue_crawler.name
policy_arn = "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole"
}
resource "aws_iam_role_policy" "glue_s3_access" {
name = "glue-s3-access"
role = aws_iam_role.glue_crawler.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = ["s3:GetObject", "s3:ListBucket", "s3:PutObject", "s3:DeleteObject"]
Resource = [
aws_s3_bucket.data_lake.arn,
"${aws_s3_bucket.data_lake.arn}/*"
]
}]
})
}CRAWL_NEW_FOLDERS_ONLY tells the crawler to only scan S3 prefixes that have not been crawled before. For partitioned data like date-based prefixes, this avoids rescanning historical data on every run and keeps crawler execution times short. Only use CRAWL_EVERYTHING when you need to detect schema changes in existing data.
delete_behavior = “LOG” means if the crawler finds that a previously cataloged table no longer exists, it logs the discrepancy rather than deleting the catalog entry. This protects against accidentally dropping table definitions when a source partition is temporarily unavailable.
Writing a Glue ETL Job
A Glue job is a PySpark or Python script that reads from one or more sources, transforms the data, and writes to a target. The Glue context wraps Spark and adds catalog awareness and job bookmarking.
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql import functions as F
from pyspark.sql.types import TimestampType
args = getResolvedOptions(sys.argv, [
"JOB_NAME",
"source_database",
"source_table",
"target_bucket",
"target_prefix"
])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)
orders_raw = glueContext.create_dynamic_frame.from_catalog(
database=args["source_database"],
table_name=args["source_table"],
transformation_ctx="orders_raw"
)
orders_df = orders_raw.toDF()
orders_cleaned = (
orders_df
.filter(F.col("order_id").isNotNull())
.filter(F.col("amount") > 0)
.withColumn("created_at", F.col("created_at").cast(TimestampType()))
.withColumn("order_date", F.to_date(F.col("created_at")))
.withColumn("year", F.year(F.col("order_date")))
.withColumn("month", F.month(F.col("order_date")))
.withColumn("day", F.dayofmonth(F.col("order_date")))
.withColumn("amount_usd", F.round(F.col("amount") / 100.0, 2))
.drop("_corrupt_record", "raw_payload")
)
orders_cleaned.write \
.mode("append") \
.partitionBy("year", "month", "day") \
.parquet(f"s3://{args['target_bucket']}/{args['target_prefix']}/")
job.commit()job.commit() at the end is what activates job bookmarking. Glue records the position of the last successfully processed data. If you rerun the job, it picks up from where it left off rather than reprocessing everything. This makes reruns safe after failures without producing duplicate records in your target.
Writing partitioned Parquet to S3 is the standard output format for a data lake. Parquet is columnar and compressed, which makes downstream Athena queries significantly cheaper. Partitioning by year, month, and day allows Athena to prune partitions and scan only the data relevant to your query.
Provisioning the Glue Job with Terraform
resource "aws_s3_object" "glue_job_script" {
bucket = aws_s3_bucket.glue_assets.bucket
key = "scripts/orders-transform.py"
source = "${path.module}/glue_scripts/orders-transform.py"
etag = filemd5("${path.module}/glue_scripts/orders-transform.py")
}
resource "aws_glue_job" "orders_transform" {
name = "orders-transform"
role_arn = aws_iam_role.glue_job.arn
glue_version = "4.0"
description = "Transforms raw order data to curated Parquet format"
command {
name = "glueetl"
script_location = "s3://${aws_s3_bucket.glue_assets.bucket}/scripts/orders-transform.py"
python_version = "3"
}
default_arguments = {
"--job-language" = "python"
"--job-bookmark-option" = "job-bookmark-enable"
"--enable-metrics" = "true"
"--enable-continuous-cloudwatch-log" = "true"
"--enable-spark-ui" = "true"
"--spark-event-logs-path" = "s3://${aws_s3_bucket.glue_assets.bucket}/spark-logs/"
"--enable-auto-scaling" = "true"
"--source_database" = aws_glue_catalog_database.raw.name
"--source_table" = "orders"
"--target_bucket" = aws_s3_bucket.data_lake.bucket
"--target_prefix" = "curated/orders"
}
execution_property {
max_concurrent_runs = 1
}
worker_type = "G.1X"
number_of_workers = 5
timeout = 60
tags = {
Environment = "production"
ManagedBy = "terraform"
}
}
resource "aws_glue_trigger" "orders_daily" {
name = "orders-daily-trigger"
type = "SCHEDULED"
schedule = "cron(0 7 * * ? *)"
actions {
job_name = aws_glue_job.orders_transform.name
}
tags = { ManagedBy = "terraform" }
}enable-auto-scaling lets Glue scale the number of workers up and down during the job based on actual parallelism demand. You set number_of_workers as a maximum. During stages of the job that are not parallelizable, workers scale down automatically. This reduces cost significantly for jobs with uneven parallelism across stages.
max_concurrent_runs = 1 prevents overlapping executions. If your scheduled trigger fires while the previous run is still in progress, the new run is skipped. For incremental jobs using bookmarking, concurrent runs would cause conflicts over the bookmark position.
Monitoring Glue Jobs
resource "aws_cloudwatch_metric_alarm" "glue_job_failed" {
alarm_name = "glue-orders-transform-failed"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = 1
metric_name = "glue.driver.aggregate.numFailedTasks"
namespace = "Glue"
period = 300
statistic = "Sum"
threshold = 0
alarm_description = "Glue orders transform job has failed tasks"
alarm_actions = [aws_sns_topic.alerts.arn]
dimensions = {
JobName = aws_glue_job.orders_transform.name
Type = "gauge"
}
}
resource "aws_cloudwatch_metric_alarm" "glue_job_duration" {
alarm_name = "glue-orders-transform-slow"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = 1
metric_name = "glue.driver.jvm.heap.usage"
namespace = "Glue"
period = 300
statistic = "Maximum"
threshold = 0.85
alarm_description = "Glue job heap usage above 85 percent, risk of OOM"
alarm_actions = [aws_sns_topic.alerts.arn]
dimensions = {
JobName = aws_glue_job.orders_transform.name
Type = "gauge"
}
}Heap usage above 85 percent is the early warning sign for out-of-memory failures. If you see this alarm fire consistently, either increase your worker type from G.1X to G.2X, reduce the partition size in your Spark job, or repartition large DataFrames before writing.
Closing Thoughts
Glue is a strong choice for teams that want serverless ETL without managing Spark infrastructure. The data catalog integration means your transformed data is immediately queryable in Athena. Job bookmarking makes incremental processing reliable. Auto-scaling keeps costs proportional to actual work done.
The patterns that matter most in production are partitioned Parquet output for query efficiency, bookmarking enabled from day one, max_concurrent_runs set to 1 for scheduled incremental jobs, and heap monitoring to catch capacity issues before they cause failures.
Enjoy the cloud.
Osama
#AWS #AWSGlue #ETL #DataEngineering #Serverless #CloudArchitecture #Terraform #InfrastructureAsCode #DataLake #Athena #ApacheSpark #AmazonWebServices #SolutionsArchitect #CloudComputing #BigData #TechBlog #CloudInfrastructure #DataPipeline #Analytics #PySpark
Technology Geek 
Leave a comment