OpenMP

Design

History

Core elements

Thread creation

Work-sharing constructs

• omp for or omp do: used to split up loop iterations among the threads, also called loop constructs.
• sections: assigning consecutive but independent code blocks to different threads
• single: specifying a code block that is executed by only one thread, a barrier is implied in the end
• master: similar to single, but the code block will be executed by the master thread only and no barrier implied in the end.

Variant directives

• The metadirective is an executable directive that conditionally resolves to another directive at compile time by selecting from multiple directive variants based on traits that define an OpenMP condition or context.
• The declare variant directive has similar functionality as metadirective but selects a function variant at the call-site based on context or user-defined conditions.

Clauses

• shared: the data declared outside a parallel region is shared, which means visible and accessible by all threads simultaneously. By default, all variables in the work sharing region are shared except the loop iteration counter.
• private: the data declared within a parallel region is private to each thread, which means each thread will have a local copy and use it as a temporary variable. A private variable is not initialized and the value is not maintained for use outside the parallel region. By default, the loop iteration counters in the OpenMP loop constructs are private.
• default: allows the programmer to state that the default data scoping within a parallel region will be either shared, or none for C/C++, or shared, firstprivate, private, or none for Fortran. The none option forces the programmer to declare each variable in the parallel region using the data sharing attribute clauses.
• firstprivate: like private except initialized to original value.
• lastprivate: like private except original value is updated after construct.
• reduction: a safe way of joining work from all threads after construct.

• critical: the enclosed code block will be executed by only one thread at a time, and not simultaneously executed by multiple threads. It is often used to protect shared data from race conditions.
• atomic: the memory update in the next instruction will be performed atomically. It does not make the entire statement atomic; only the memory update is atomic. A compiler might use special hardware instructions for better performance than when using critical.
• ordered: the structured block is executed in the order in which iterations would be executed in a sequential loop
• barrier: each thread waits until all of the other threads of a team have reached this point. A work-sharing construct has an implicit barrier synchronization at the end.
• nowait: specifies that threads completing assigned work can proceed without waiting for all threads in the team to finish. In the absence of this clause, threads encounter a barrier synchronization at the end of the work sharing construct.

• schedule: This is useful if the work sharing construct is a do-loop or for-loop. The iteration in the work sharing construct are assigned to threads according to the scheduling method defined by this clause. The three types of scheduling are:

1. static: Here, all the threads are allocated iterations before they execute the loop iterations. The iterations are divided among threads equally by default. However, specifying an integer for the parameter chunk will allocate chunk number of contiguous iterations to a particular thread.
2. dynamic: Here, some of the iterations are allocated to a smaller number of threads. Once a particular thread finishes its allocated iteration, it returns to get another one from the iterations that are left. The parameter chunk defines the number of contiguous iterations that are allocated to a thread at a time.
3. guided: A large chunk of contiguous iterations are allocated to each thread dynamically. The chunk size decreases exponentially with each successive allocation to a minimum size specified in the parameter chunk

• if: This will cause the threads to parallelize the task only if a condition is met. Otherwise the code block executes serially.

• firstprivate: the data is private to each thread, but initialized using the value of the variable using the same name from the master thread.
• lastprivate: the data is private to each thread. The value of this private data will be copied to a global variable using the same name outside the parallel region if current iteration is the last iteration in the parallelized loop. A variable can be both firstprivate and lastprivate.
• threadprivate: The data is a global data, but it is private in each parallel region during the runtime. The difference between threadprivate and private is the global scope associated with threadprivate and the preserved value across parallel regions.

• copyin: similar to firstprivate for private variables, threadprivate variables are not initialized, unless using copyin to pass the value from the corresponding global variables. No copyout is needed because the value of a threadprivate variable is maintained throughout the execution of the whole program.
• copyprivate: used with single to support the copying of data values from private objects on one thread to the corresponding objects on other threads in the team.

• reduction: the variable has a local copy in each thread, but the values of the local copies will be summarized into a global shared variable. This is very useful if a particular operation on a variable runs iteratively, so that its value at a particular iteration depends on its value at a prior iteration. The steps that lead up to the operational increment are parallelized, but the threads updates the global variable in a thread safe manner. This would be required in parallelizing numerical integration of functions and differential equations, as a common example.

• flush: The value of this variable is restored from the register to the memory for using this value outside of a parallel part
• master: Executed only by the master thread. No implicit barrier; other team members not required to reach.

  User-level runtime routines

Environment variables

Implementations

• GCC 4.3.1
• Mercurium compiler
• Intel Fortran and C/C++ versions 11.0 and 11.1 compilers, Intel C/C++ and Fortran Composer XE 2011 and Intel Parallel Studio.
• IBM XL compiler
• Sun Studio 12 update 1 has a full implementation of OpenMP 3.0

• GCC 4.7
• Intel Fortran and C/C++ compilers 12.1
• IBM XL C/C++ compilers for AIX and Linux, V13.1 & IBM XL Fortran compilers for AIX and Linux, V14.1
• LLVM/Clang 3.7
• Absoft Fortran Compilers v. 19 for Windows, Mac OS X and Linux

• GCC 4.9.0 for C/C++, GCC 4.9.1 for Fortran
• Intel Fortran and C/C++ compilers 15.0
• IBM XL C/C++ for Linux, V13.1 & XL Fortran for Linux, V15.1
• LLVM/Clang 3.7

• GCC 6 for C/C++
• Intel Fortran and C/C++ compilers 17.0, 18.0, 19.0

• iPat/OMP
• Parallware
• PLUTO
• ROSE
• S2P by KPIT Cummins Infosystems Ltd.

• Allinea Distributed Debugging Tool – debugger for OpenMP and MPI codes
• Allinea MAP – profiler for OpenMP and MPI codes
• TotalView - debugger from Rogue Wave Software for OpenMP, MPI and serial codes
• ompP – profiler for OpenMP
• VAMPIR – profiler for OpenMP and MPI code

  Pros and cons

• Portable multithreading code.
• Simple: need not deal with message passing as MPI does.
• Data layout and decomposition is handled automatically by directives.
• Scalability comparable to MPI on shared-memory systems.
• Incremental parallelism: can work on one part of the program at one time, no dramatic change to code is needed.
• Unified code for both serial and parallel applications: OpenMP constructs are treated as comments when sequential compilers are used.
• Original code statements need not, in general, be modified when parallelized with OpenMP. This reduces the chance of inadvertently introducing bugs.
• Both coarse-grained and fine-grained parallelism are possible.
• In irregular multi-physics applications which do not adhere solely to the SPMD mode of computation, as encountered in tightly coupled fluid-particulate systems, the flexibility of OpenMP can have a big performance advantage over MPI.
• Can be used on various accelerators such as GPGPU and FPGAs.

• Risk of introducing difficult to debug synchronization bugs and race conditions.
• only runs efficiently in shared-memory multiprocessor platforms.
• Requires a compiler that supports OpenMP.
• Scalability is limited by memory architecture.
• No support for compare-and-swap.
• Reliable error handling is missing.
• Lacks fine-grained mechanisms to control thread-processor mapping.
• High chance of accidentally writing false sharing code.

  Performance expectations

• When a dependency exists, a process must wait until the data it depends on is computed.
• When multiple processes share a non-parallel proof resource, their requests are executed sequentially. Therefore, each thread must wait until the other thread releases the resource.
• A large part of the program may not be parallelized by OpenMP, which means that the theoretical upper limit of speedup is limited according to Amdahl's law.
• N processors in a symmetric multiprocessing may have N times the computation power, but the memory bandwidth usually does not scale up N times. Quite often, the original memory path is shared by multiple processors and performance degradation may be observed when they compete for the shared memory bandwidth.
• Many other common problems affecting the final speedup in parallel computing also apply to OpenMP, like load balancing and synchronization overhead.
• Compiler optimisation may not be as effective when invoking OpenMP. This can commonly lead to a single-threaded OpenMP program running slower than the same code compiled without an OpenMP flag.

  Thread affinity

Benchmarks

• a collection of applications that allow to test OpenMP tasking implementations.
• SPEC series
• *
• * testing OpenMP 4 target offloading API
• *
• focusing on accelerators.

• is a benchmark suite designed to systematically and quantitatively evaluate the effectiveness of OpenMP data race detection tools.
• is a benchmark suite to evaluate compilers and tools which can automatically insert OpenMP directives.