Benchmarking NVIDIA RAPIDS on GKE: Estimated vs. Actual Spark Speedups

I turned on the NVIDIA RAPIDS Accelerator for a set of Spark SQL and PySpark jobs on Google Kubernetes Engine and expected those jobs to finish several times faster. The Qualification Tool agreed. It placed several workloads in the Large speedup category and estimated a 4–5× GPU speedup.

I then compared end-to-end wall-clock time with the same jobs on CPU. Some of the RAPIDS runs finished slower.

This is the gap I care about: what that tool is scoring, where the extra minutes went on GKE, and the three rules I kept after the benchmarks. The manifests, the qualification wrapper, and the full configuration notes live in here. Copy the YAML from there. The rest of this piece is why I stopped treating GPU Spark as a cluster-wide default.

The stack I actually ran

The deployment was Apache Spark 3.5.5 with RAPIDS Accelerator 25.12.0 on CUDA 12. Executors landed...

Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE