\section{Conclusion}

The ThreadPool library within Trilinos provides a simple, minimalistic API for HPC applications to effectively use hybrid parallelism on HPC systems with CPU-based manycore nodes.
%
The ThreadPool library is currently implemented in the standard C language using the standard \textbf{pthread} library.
%
The ThreadPool library creates a pool of \emph{worker} threads which are held ready for use by the application.
%
The ThreadPool API assumes an application programming model
(Figure~\ref{fig:HybridParallelArchitecture}) which separates its software into lower-level stateless computational kernels and higher-level control flow and resource management components.
%
ThreadPool functions are called by an application to run computational kernels thread-parallel.
%
Inter-thread synchronization is supported through mutually exclusive execution locks, or through more efficient reduction operations.



A simple mini-application performing conjugate gradient solution algorithm iterations was run on Sandia National Laboratories' \emph{glory} cluster with 272 quad-socket / quad-core compute nodes with non-uniform memory access (NUMA).
%
This mini-application on this cluster demonstrated significantly improved performance, for CPU cache-resident problems, by nesting thread-parallelism within CPU-cores, while retaining MPI-parallelism between CPU-sockets.
%
For large problems with performance limited by access to main memory the performance improvement is relatively small.
%
These results show that hybrid thread-parallelism nested within MPI-parallelism can improve HPC application performance when run on clusters of multicore compute nodes.



\subsection{To-be-done: Comparison other Threading Capabilities}

Hybrid parallel performance results presented in this report use standard pthreads managed by the ThreadPool interface.
%
Other thread parallel mechanisms such as OpenMP and TBB could yield different performance results.
%
The hybrid parallel conjugate gradient performance test cases could be re-implemented using OpenMP and TBB to compare and evaluate the ThreadPool library's efficiency and API compared to these other threading capabilities.



 
