Parallel Computing

Before you dive into this, let me just tell you the punchline of this entire page right up front: parallelism is the last tool you want to turn to for speed. It is not a silver bullet, it will generally take you significant time to implement, the speed improvements from parallelism are generally much smaller than what you get from other performance improvement methods (see Understanding Performance and Performance Solutions), and the headaches of parallelizing code are many.

What is Parallelism

Why is Paralleism important?

Moore’s law kinda is dead! see: single thread performance

images/42-years-processor-trend.png

Source: https://www.karlrupp.net/2018/02/42-years-of-microprocessor-trend-data/

Further reading / resources:

https://www.youtube.com/watch?v=zX4ZNfvw1cw

Limits of Parallelism

Most task are at least partially serial, and that limits speed up. If your program is 90% parallelizable, all the cores in the world will only get you a 10x speed-up!

Amdahl’s Law

\[\frac{1}{(1-P)+\frac{P}{N}}\]

Types of Parallelism

Multi-processing

Multi-threading

DANGER! HERE BE DRAGONS! RACE CONDITIONS!