# AWS Glue Silent Failures: A 3-Day Debugging Story Nobody Talks About Let me tell you about the day I spent 72 hours debugging an AWS Glue job that wasn't actually broken. The story started simple. We had a Glue ETL job running on schedule in production. Raw data from S3, transformation using PySpark, write back as Parquet. Downstream, Amazon Athena would query the output for business intelligence dashboards. The job worked fine in dev. Ran perfectly for two weeks in production. Then one Tuesday afternoon it just... failed. No error message. No stack trace. Just **"Job run failed"** in the Glue console. This is the problem with Glue. When it fails, it fails *quietly*. --- ## The Debugging Journey (Or: Why I Learned to Hate Logs) ### Step 1: Blame Airflow (Wrong) Our first instinct was to check the Apache Airflow logs. We use MWAA (Managed Workflows for Apache Airflow) to orchestrate the Glue job. ``` [2024-01-15 14:32:15] Task failed with error: ClientError: An error occurred (InvalidInputException) when calling the RunJob operation: The supplied role ARN is invalid. ``` Except... the role ARN was the same one we've been using for two weeks. So why now? Checked IAM permissions. Looked fine. Restarted the MWAA task. Same error. Wasted 4 hours here. ### Step 2: Go to the Glue Console (Still Wrong) Opened the AWS Glue job run details page. This is where it gets frustrating. The console showed: ``` Job Status: FAILED Error: Job run failed Reason: See logs for details Last Update: 2 minutes ago ``` That's it. That's the entire error message. "See logs for details." Okay, Glue. Let's go to the logs then. ### Step 3: CloudWatch Logs Hell (Where the Real Story Starts) Here's what Glue doesn't tell you upfront: **It logs to THREE different CloudWatch log groups simultaneously**, and the error is almost never in the obvious one. The three log groups are: - `/aws-glue/jobs/output` — Driver logs (usually useless for actual errors) - `/aws-glue/jobs/error` — This is where you want to be - `/aws-glue/jobs/glue-python-shell` — Only for Python Shell jobs (we're using Spark, not relevant) I found the error logs buried under `/aws-glue/jobs/error` in an executor log stream (not the driver). The job had **multiple workers**, and each worker had its own log stream. The error was in worker 3 of 4. ``` Worker-3 - ERROR - Executor 5: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3145) at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.compile(CodeGenerator.scala:1069) ... [20 more stack traces] ``` Found it. **Out of memory**. A single worker ran out of RAM. ### Step 4: Why is One Worker Out of Memory? (The Real Problem) Dug deeper. The job was processing one day's worth of data. But one specific date partition had **10 gigabytes of data**. All other partitions? 800 megabytes to 1.2 gigabytes. That one huge partition was assigned to a single worker. Worker memory? Default 2 GB per node. Why was the partition so big? A month ago, someone ran a manual backfill operation. They loaded historical data and didn't partition it correctly. It all went into one date bucket. But wait, there was more. The job was using the **default number of DPUs**. For context, AWS Glue measures compute in "Data Processing Units" (DPUs). Default is 2 DPUs = 1 driver + 1 worker. We were processing production data with the compute equivalent of a laptop from 2015. ### Step 5: Secondary Issue — Job Bookmarks Weren't Enabled While reading the Glue job configuration, I noticed job bookmarks were disabled. This meant every time the job ran, it reprocessed ALL data, not just new data. So on that Tuesday: 1. Default 2 DPU (1 driver + 1 worker) 2. Worker got the huge partition 3. Worker spilled memory to disk 4. Disk fills up 5. Job times out 6. No clear error in the console This is the silent failure problem. The error happened at the executor level, not the driver level. The driver says "uh, I lost connection to an executor" and then gives up. --- ## The Fix (And Why It Should Have Been Done Day One) ### Fix 1: Increase DPU Count ```python # Before: Default 2 DPU # After: 5 DPU = 1 driver + 4 workers cluster_config = { "Name": "production-etl-glue-job", "NumberOfWorkers": 4, "WorkerType": "G.2X", # General purpose, 8GB RAM each "GlueVersion": "4.0", "MaxCapacity": None, # Using WorkerType, not MaxCapacity "Timeout": 60, # 60 minutes, up from default 30 } ``` More workers = data distributed across more machines = no single worker drowns. ### Fix 2: Repartition Before Writing ```python # In the Spark transformation df = spark.read.parquet("s3://raw-data/") # Transform... transformed_df = df.filter(...).select(...).groupBy(...).agg(...) # CRITICAL: Repartition by date before writing # This ensures even distribution, not by accident repartitioned_df = transformed_df.repartition(32, "date_column") # Write with partitioning repartitioned_df.write \ .mode("overwrite") \ .partitionBy("year", "month", "day") \ .parquet("s3://processed-data/") ``` The `repartition()` call redistributes data across workers. Even if your source data is uneven, you're redistributing it equally before the write. ### Fix 3: Enable Job Bookmarks ```python # Glue job config { "JobBookmarkOption": "job-bookmark-enable", "MaxRetries": 1, "Timeout": 60, # ... rest of config } ``` Job bookmarks track which data has been processed. Next run only processes new data. Saves time and compute costs. ### Fix 4: CloudWatch Alarms (The Real Hero) ```python # In your Terraform or CloudFormation resource "aws_cloudwatch_metric_alarm" "glue_job_failures" { alarm_name = "glue-job-production-etl-failures" comparison_operator = "GreaterThanOrEqualToThreshold" evaluation_periods = "1" metric_name = "glue_job_run_failure" namespace = "AWS/Glue" period = "300" statistic = "Sum" threshold = "1" alarm_description = "Alert when Glue job fails" alarm_actions = [aws_sns_topic.alerts.arn] } ``` Instead of discovering failures from Athena dashboards going stale, you get paged immediately. --- ## What I Wish I'd Known (Lessons That Stuck) ### 1. **Executor Logs Are Where the Truth Lives** Driver logs are garbage. They don't see what workers are doing. The real error? Always in executor logs, always harder to find. *Lesson:* First thing when Glue fails: go to CloudWatch, find `/aws-glue/jobs/error` log group, search for "ERROR" in executor log streams (not driver). ### 2. **Default Anything in Production is Wrong** Default 2 DPU worked for toy datasets. Real production data laughs at defaults. *Lesson:* Measure your data size. Do the math. 50 GB of data? You need at least 10-15 DPU. Let Glue estimate then add 40% overhead. ### 3. **Partition Skew is a Silent Killer** One huge partition and three normal ones? That one partition becomes your bottleneck. One worker drowns while three others are idle. *Lesson:* Always inspect your partition sizes in S3. Look at the distribution. If something is 10x larger, investigate. ### 4. **Job Bookmarks Should Be Day One, Not Day 30** Reprocessing all data every run is wasteful. But more importantly, it makes failures more likely because you're processing way more data than necessary. *Lesson:* Enable bookmarks before your first production run. Yes, it complicates debugging locally. It saves your skin in production. ### 5. **Set CloudWatch Alarms From Day One** We discovered the Glue failure from a Slack bot that checks Athena table freshness. That's a **human detecting a machine failure**. Backwards. *Lesson:* Alarm on Glue job failures before anything else alarms on missing data. --- ## The Time Cost - **Step 1 (Airflow logs):** 4 hours - **Step 2 (Glue console):** 2 hours - **Step 3 (Finding executor logs):** 8 hours - **Step 4 (Root cause analysis):** 12 hours - **Step 5 (Fix and test):** 46 hours **Total: 3 days for a fix that was 20 minutes once we knew what to fix.** That's not a failure of Glue. That's a failure of observability. We had no way to see what was happening at the executor level until we manually opened CloudWatch logs and scrolled through thousands of lines. --- ## The Reality of AWS Glue Glue is solid. It's been battle-tested by thousands of companies. But it's designed for when things work. When they break, it's like debugging in the dark. The error handling is... minimal. The monitoring is.. optional. The logging is... verbose but disorganized. Here's what you need to know: **Glue does:** - ✅ Run PySpark/Scala transformations on scale - ✅ Handle variable data volumes (if you size it right) - ✅ Integrate with S3, Redshift, RDS, Athena - ✅ Schedule jobs reliably (most of the time) **Glue doesn't:** - ❌ Tell you what went wrong when it fails - ❌ Warn you about upcoming memory issues - ❌ Suggest DPU count fixes - ❌ Default to sensible production settings You have to do that part. --- ## What to Do Monday Morning If you're running Glue jobs in production, do this: 1. **Increase DPU to 5+** if your data is more than 10 GB per run 2. **Enable job bookmarks** immediately 3. **Set CloudWatch alarms** on job failures 4. **Check partition sizes** in S3 — if anything is 5x larger than others, investigate 5. **Add `.repartition()` before writes** to ensure even distribution The 72 hours I spent debugging would have been 30 minutes if I'd done these things upfront. Don't learn this lesson the hard way. Learn it from someone who did. --- ## Questions You'll Ask **Q: Will increasing DPU to 5 cost a fortune?** A: DPU costs ~$0.44/DPU per hour on-demand. 5 DPU × 1 hour = ~$2.20. Daily job. ~$66/month extra if run daily. Worth it to avoid 3-day debugging sessions. **Q: Can I use Spark repartition() without running out of memory?** A: Not if you're already out of memory. Repartition shuffles data across the network. If you're hitting memory limits, do repartition BEFORE the transformation that's causing memory issues, not after. **Q: Job bookmarks broke my historical reprocessing. What now?** A: Bookmarks have a reset option. You can disable it for a run, or reset the bookmark state. Use it during backfills, then re-enable. **Q: Should I use Databricks instead?** A: Different problem. Databricks is SQL + ML platform. Glue is ETL orchestration. Different products, different use cases. This article is about Glue in production and making it not fail silently. --- **Have you hit Glue failures before? What did your error message actually say?** Drop it in the comments. Bet you anything it was unhelpful and you spent 12 hours finding the real issue.