High Performance Spark

Master Spark Performance at Scale

Apache Spark is the dominant framework for large-scale data processing, but writing fast Spark code requires deep knowledge of its internals, execution model, and configuration. This book gives you that knowledge.

Now fully updated for Apache Spark 4, this second edition covers the latest APIs, performance improvements, and best practices. Whether you're optimizing existing pipelines, designing new ones from scratch, or tuning a production cluster, High Performance Spark provides battle-tested patterns and practical advice grounded in real-world experience.

• How Spark's execution model, DAG scheduler, and memory management work
• When to use RDDs, DataFrames, Datasets, and Spark SQL — and why it matters
• MLlib and machine learning performance patterns
• Efficient joins, shuffles, and data skew mitigation strategies
• Serialization, caching, and memory tuning for large workloads
• Resource management (including accelerators like GPUs), and effective dynamic allocation
• Testing, monitoring, and debugging Spark applications
• Integration with modern data lakehouse patterns, with a focus on Iceberg
• Python (PySpark) and Scala performance trade-offs
• Structured Streaming best practices for low-latency pipelines (completely new in this edition)