Code Quiz

Coalesce Won't Increase Spark Partitions

A subtle Spark bug where coalesce fails to increase parallelism before a scaling join.

Codepython

from pyspark.sql import functions as F
from pyspark.sql.functions import broadcast

# Source table read from a few large files -> ~8 input partitions
df = spark.read.parquet("s3://events/raw")

# Heavy filter drops ~90% of rows but keeps skewed key distribution
df = df.filter(F.col("active") == True)

# We want MORE parallelism for the feature-scaling stage below,
# so bump partitions up to 400 before the join.
df = df.coalesce(400)

# Per-key scaling stats to standardize the 'amount' feature
stats = (df.groupBy("user_id")
           .agg(F.mean("amount").alias("mu"),
                F.stddev("amount").alias("sd")))

scaled = (df.join(broadcast(stats), on="user_id")
            .withColumn("amount_z", (F.col("amount") - F.col("mu")) / F.col("sd")))

scaled.write.parquet("s3://events/scaled")

The scaling stage stays slow and single-threaded despite requesting 400 partitions. What is the bug?

Next →Distributed Feature Scaling and Shuffles in Spark