Grokking Parallel Programming (With Examples in CUDA)

Scientific simulations, graphics-intensive games, and, of course, AI models, all require the serious computing power delivered by GPUs (graphics processing units). This book teaches you to write parallel GPU code from scratch using CUDA, NVIDIA’s platform for general-purpose GPU computing. Perfect for readers with no experience in parallel programming or CUDA, this book sets out a clear path with well-annotated code, helpful diagrams, and complete, easy-to-understand explanations.

Written by SungHwan Yun, a former Samsung Principal Engineer and competitive programming problem setter, this friendly, fully illustrated guide takes you from basic definitions through your very first parallel kernel to the memory-optimization techniques that turn correct code into fast code. Chapter by chapter, you’ll explore the fundamental patterns of parallel programming from the ground up—all with examples in CUDA that make it very real.

You’ll start with something super simple: doubling the elements of an array in parallel. Then, you’ll see how this core idea leads all the way up to summing millions of numbers with tournament-style reductions, filtering data with parallel prefix sums, and speeding up matrix multiplication with shared-memory tiling. And the examples get more interesting as you go! You’ll see how parallel programming applies to graphics, data visualization, and accelerating AI training and inference.

Each hands-on chapter ends with a section that walks you through the most common bugs—such as race conditions, missing barriers, and out-of-bounds access—so you can easily spot and fix them in your own code. All the examples are grounded in CUDA, Nvidia’s platform that lets developers use graphics cards for general math and data work. By the end, you’ll be thinking like a GPU engineer, ready to take on real-world parallel programming tasks!

And if you don’t have access to a GPU, no problem! Grokking Parallel Programming includes full access to a free online sandbox that lets you write CUDA code in your browser, submit it, and get an instant verdict—no hardware required. The practice problems are designed with the rigor of a veteran competitive programming problem setter, each targeting one specific concept with carefully constructed test data.

What's inside

• No experience with CUDA or GPUs required!
• Write, launch, and debug CUDA kernels from scratch
• Parallel patterns: map, reduction, scan, histogram, and stencil
• Free access to an online GPU environment

About the reader

For software engineers, machine learning practitioners, students, and self-learners who are comfortable with basic C and want to add GPU programming to their toolkit. No prior parallel programming or GPU experience is required.

About the author

SungHwan Yun spent 17 years at Samsung Electronics as a Principal System Software Engineer, where he led Linux kernel development for the Galaxy smartphone series. He is a recognized leader in competitive problem solving, ranked #9 nationally with 9,700+ problems solved, and he contributed 132 original problems as the #2 problem setter at Baekjoon Online Judge.