Introduction to Parallel Programming

26 April 2021

Obed N Munoz

Cloud Software Engineer

Concurrent vs Parallel


Parallel Programming - Serial Computing


Parallel Programming - Parallel Computing


Parallel Programming - Parallel Computers 1/2

Stand-alone Computers
- Multiple functional units (Lx Caches, prefetch, decode, floating pont, GPU, etc)
- Multiple execution units/cores
- Multiple hardware threads


Parallel Programming - Parallel Computers 2/2

Multiple stand-alone computers to make a larger parallel computer(cluster).


Parallel Computing - Where?


Parallel Computing - Why?


Jargon of Parallel Programming

von Neumann Architecture

So what? Who cares?
Parallel computers still follow this basic design, just multiplied in units. The basic, fundamental architecture remains the same.


Jargon of Parallel Programming - Terms


Flynn's Taxonomy

Distinguishes multi-processor computer architectures according to how they can be classified along the two independent dimensions of Instruction Stream and Data Stream. Each of these dimensions can have only one of two possible states: Single or Multiple.


Parallel Architectures - Shared Memory

Uniform Memory Access (UMA)


Parallel Architectures - Shared Memory

Non-Uniform Memory Access (NUMA)


Parallel Architectures - Distributed Memory

Distributed memory systems require a communication network to connect inter-processor memory.


Parallel Architectures - Hybrid Distributed-Shared Memory

The largest and fastest computers in the world today employ both shared and distributed memory architectures.


Parallel Programming Pattern's Definition

In the book of Patterns for Parallel Programming from Massingill, Sanders and Mattson there's a pattern language that helps on the process of understanding and designing parallel programs.


Finding Concurrency

Programmers should start their design of a parallel solution by analyzing the problem within the problem domain to expose exploitable concurrency.

Is the problem large enough and the results significant enough to justify?


Algorithm Structure

Our goal is to refine the design and move it closer to a program that can execute tasks concurrently by mapping the concurrency onto multiple UEs running on a parallel computer.

The key issue at this stage is to decide which pattern or patterns are most appropriate for the problem.


Supportting Structures

We call these patterns Supporting Structures because they describe software constructions or "structures" that support the expression of parallel algorithms.


Implementation mechanisms

Every parallel program needs to:

1. Create the set of UEs.
2. Manage interactions between them and their access to shared resources
3. Exchange information between UEs.
4. Shut them down in an orderly manner.

Parallel Programming Models

More details at:


Parallel Programming Design Considerations

More at:


Resources and Credits

This material is genereated thanks to some extracts from following resources:


Thank you

Obed N Munoz

Cloud Software Engineer

Use the left and right arrow keys or click the left and right edges of the page to navigate between slides.
(Press 'H' or navigate to hide this message.)