IMP: Integrative Model for Parallelism

The Integrative Model for Parallelism is a new development in parallel programming. It allows
for high level expression of parallel algorithms, giving efficient execution in multiple parallelism modes.

Elevator pitch

The type of parallelism that is used in scientific computing is of a type that can be described as `generalized data parallelism’.
For instance a power method multiplies a matrix times a vector, inner product of the output vector and the input,
use that scalar to scale the output vector, et cetera.
This is a sequential program: the parallelism stems from the fact that the objects are spread of a large number of processors.
So a programmer should be able to write in these sequential terms, maybe indicating data distributions, but
without explicit concern for the communication and synchronization between processors.

Such `sequential semantics’ languages have been tried before, but without much success, mostly because
of a lack of performance. The reason for this is that the compiler and runtime system can not relate
the algorithm to the data distributions. The Integrative Model for Parallelism (IMP)
has effected a separation of concerns where specifying the distribution (independent of the algorithm)
and the so-called signature of a data parallel function (which is independent of the parallelism)
together allow the system to derive
the full communication and synchronization structure of an algorithm.

The IMP system translates an algorithm in sequential semantics to a DAG-like representation, in which the
communication and synchronization mechanisms are not yet made specific. That means that the relations in this DAG
can be realized through message passing, task dependencies, or accelerator offloading.

Prototype tests show that IMP codes perform comparable to hand-written message passing codes.

I was interviewed on InsideHPC

A gentle introduction to IMP

The mathematical basis for IMP

progress-report lecture at SC15

Latency hiding in the Integrative Model

Repository of code and technical reports


Eijkhout, V.
A DSL for Integrative Parallel Programming
Parallel and Distributed Computing (ISPDC), 2014 IEEE 13th International Symposium on, 2014, 27-34

Eijkhout, V.
A Theory of Data Movement in Parallel Computations
Procedia Computer Science, 2012, 9, 236 - 245