%@HEADER
// ***********************************************************************
// 
//                Amesos: Direct Sparse Solver Package
//                 Copyright (2004) Sandia Corporation
// 
// Under terms of Contract DE-AC04-94AL85000, there is a non-exclusive
// license for use of this work by or on behalf of the U.S. Government.
// 
// This library is free software; you can redistribute it and/or modify
// it under the terms of the GNU Lesser General Public License as
// published by the Free Software Foundation; either version 2.1 of the
// License, or (at your option) any later version.
//  
// This library is distributed in the hope that it will be useful, but
// WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
// Lesser General Public License for more details.
//  
// You should have received a copy of the GNU Lesser General Public
// License along with this library; if not, write to the Free Software
// Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301
// USA
// Questions? Contact Michael A. Heroux (maherou@sandia.gov) 
// 
// ***********************************************************************
%@HEADER

\documentclass[11pt]{SANDreport}
\usepackage{graphicx}
\usepackage{amsmath,amsfonts,amsthm}
\usepackage{amssymb,latexsym}
\usepackage{rotating,tabularx}

\usepackage{times}

\def\choicebox#1#2{\noindent$\hphantom{th}$\parbox[t]{2.10in}{\sf
#1}\parbox[t]{3.35in}{#2}\\[0.8em]}

\setlength\oddsidemargin {-0pt}\setlength\evensidemargin{-0pt}
\setlength{\textwidth}{150mm}\setlength{\textheight}{210mm}
\headheight=12.45pt


\title{Amesos 2.0 Reference Guide}
\SANDnum{SAND2004-4820}
\SANDauthor{
Marzio Sala
}

\SANDprintDate{September 2004 (last updated on Jan 2006)}
\SANDreleaseType{Unlimited Release}

\newcommand{\epetra}{{\sc Epetra}}
\newcommand{\amesos}{{\sc Amesos}}
\newcommand{\teuchos}{{\sc Teuchos}}
\newcommand{\ml}{{\sc ML}}
\newcommand{\ifpack}{{\sc Ifpack}}
\newcommand{\triutils}{{\sc Triutils}}
\newcommand{\trilinos}{{\sc Trilinos}}
\newcommand{\TrilinosTM}{Trilinos \copyright}

\newtheorem{remark}{Remark}

\begin{document}

\maketitle

\begin{center}
Marzio  Sala \\
Computational Math \& Algorithms \\
Sandia National Laboratories\\
P.O.~Box 5800, MS 1110 \\
Albuquerque, NM 87185-1110
\end{center}

\vspace{20pt}

\begin{abstract}
  This document describes the main functionalities of the version 2.0 of the
  \amesos\ package. \amesos\ provides an
  object-oriented interface to several serial and parallel sparse direct
  solvers libraries for the solution of the linear system of equations
  \begin{equation}
    \label{eq:linear_system}
    A X = B,
  \end{equation}
  where $A$ is a real sparse, distributed matrix, defined as an
  Epetra\_RowMatrix object, and $X$ and $B$ are defined as
  Epetra\_MultiVector objects.   \amesos\ provides a common look-and-feel
  for all interfaces, and insulates the user from each solver's details, such as matrix and
  vector formats, and data distribution. Currently supported libraries are:
  LAPACK, KLU, UMFPACK, PARDISO, TAUCS, SuperLU, SuperLU\_DIST, MUMPS, DSCPACK.
  
  \smallskip

  This document is organized as follows. First, Section~\ref{sec:intro}
  introduces the design of \amesos.  Section~\ref{sec:basic} presents
  the basic usage of the \amesos\ package. Section~\ref{sec:configuration}
  details how to configure and compile \amesos.
  Section~\ref{sec:classes} describes the interfaces of \amesos to the supported
  direct solvers. A brief note on
  the examples included in the distribution is reported in Section~\ref{sec:examples}.
\end{abstract}

\clearpage

\clearpage

%-----------------------------------------------------------------------------
\section*{Acknowledgments}
%-----------------------------------------------------------------------------
The author would like to acknowledge the support of the ASCI and LDRD programs
that funded development of \amesos.

\clearpage

\SANDmain


\tableofcontents

\clearpage

%-----------------------------------------------------------------------------
\section{The Design of \amesos}
\label{sec:intro}
%-----------------------------------------------------------------------------

The \amesos\ package, developed by (in alphabetical order) T. Davis, 
M. Heroux, R. Hoekstra, M. Sala, and K. Stanley, is an effort to define a set
of object-oriented, abstract interfaces for the usage of serial and 
parallel sparse direct solvers. Although one serial direct solver, KLU, 
  is distributed within \amesos, the goal of the \amesos\ project is to make
  it easier to interface a code that makes use of \epetra\ objects with
  direct solver libraries developed outside \trilinos.

\amesos\ is written in C++, and has been designed with the
following requirements:
\begin{itemize}
\item {\bf Simplicity of usage:} Solving linear system (\ref{eq:linear_system}) in a language
like MATLAB is very easy, just write \verb!X = A \ b!. It should not be much
more difficult in a C++, production code.
\item {\bf Flexibility:} More than one algorithm must be available, 
  to offer optimal algorithms for small and large matrices, serial and
  parallel.
\item {\bf Efficiency:} The solution of (\ref{eq:linear_system}) must be as
efficient as possible, using state-of-the-art algorithms. Besides, the
overhead due to the C++ design must be minimal.
\end{itemize}

To fulfill these design requirements, we split the solution
of linear system (\ref{eq:linear_system}) into the
following steps:
\begin{enumerate}
\item Definition of the sparsity pattern of the linear system matrix;
\item Computation of the symbolic factorization;
\item Definition of the values of the linear system matrix;
\item Computation of the numeric factorization;
\item Definition of the values of the right-hand side;
\item Solution of the linear system.
\end{enumerate}
Steps 1, 3 and 5 depend on the matrix and vector format. To increase
flexibility, \amesos\ requires the
matrix to be derived from the \verb!Epetra_RowMatrix! format, and the solution
and right-hand side vector to the defined as \verb!Epetra_MultiVector!'s.

Steps 2, 4 and 6 correspond to three different methods in the \amesos\ classes.
From an abstract point of view, these steps do not depend on the direct
solver of choice. However, their concrete implementations does, since different
libraries may require different matrix and vector formats and distribution,
may have different parameters or different ways of setting the same parameter.
To obtain flexibility, \amesos\ insulates 
the user from the details specific to each solver, so that generic
methods can be used to manipulate all the supported interface. This design
goal is accomplished using a pure virtual class, which defines
methods \verb!SymbolicFactorization()!, \verb!NumericFactorization()! and
\verb!Solve()!, plus method \verb!SetParameters()! which can be used to tune
the interface.

To increase efficiency, all \amesos\ classes are defined as light containers.
Each class simply converts the matrix \verb!A! from the input
\verb!Epetra_RowMatrix! format into the solver's required format, and sets the
parameters are defined by the user. Therefore, \amesos\ interfaces are as
efficient as the underline solver library.

%-----------------------------------------------------------------------------
\section{Basic Usage}
\label{sec:basic}
%-----------------------------------------------------------------------------

A fragment of code using \amesos\ is as follows.
Let us suppose that \verb!A! is an \verb!Epetra_RowMatrix!, and \verb!X! and
\verb!B! are two \verb!Epetra_MultiVector!'s. 
First,
  we need to include the header files for \amesos:
\begin{verbatim}
#include "Amesos.h"
#include "Amesos_BaseSolver.h"
\end{verbatim}
Note that these header files will {\sl not} include the header files for the
supported libraries (which are of course needed to compile the \amesos\
library itself).  Then, we need to create an linear problem, as follows:
\begin{verbatim}
Epetra_LinearProblem Problem(&A, &X, &B);
\end{verbatim}
At this point, we can create an \amesos\ class using the factory class
\verb!Amesos!:
\begin{verbatim}
Amesos_BaseSolver* Solver;
Amesos Factory;
char* SolverType = "Amesos_Klu"; // uses the KLU direct solver
Solver = Factory.Create(SolverType, Problem);
\end{verbatim}
At this point, we can perform the symbolic factorization
of the linear system matrix:
\begin{verbatim}
AMESOS_CHK_ERR(Solver->SymbolicFactorization());
\end{verbatim}
This phase does not require the numerical values of \verb!A!, which can
therefore be changed after the call to \verb!SymbolicFactorization()!.
However,  the nonzero pattern of \verb!A! {\em cannot} be
changed.
\verb!AMESOS_CHK_ERR! is a macro (defined in \verb!Amesos_ConfigDefs.h!)
that checks the return code: if not zero, the
macro prints out an error message, and returns.
The numeric factorization is performed by
\begin{verbatim}
AMESOS_CHK_ERR(Solver->NumericFactorization());
\end{verbatim}
\verb!NumericFactorization()! accesses the values of \verb!A!, but does not
consider the vectors \verb!X! and \verb!B!. Finally, to solve the linear
system, we simply write
\begin{verbatim}
AMESOS_CHK_ERR(Solver->Solve());
\end{verbatim}

In the previous example, we showed how to use the KLU solver 
(see Section~\ref{sec:klu} for more details). Other interfaces
can be created using the factory class by simply changing one parameter. Note
that the supported solver can be serial or parallel, dense or sparse: the user
code still remains the same (except for the name of the solver); \amesos\ will
take care of data redistribution if required by the selected solver. The list
of supported solvers is reported in Table~\ref{tab:classes}. 
 Method
\verb!Factory.Query()! can be used to query the factory about the
availability of a given solver:
\begin{verbatim}
char* SolverType = "Amesos_Klu";
bool IsAvailable = Factory.Query(SolverType);
\end{verbatim}

\begin{table}[tbhp]
  \centering
  \begin{tabular}{| l | l l | l | }
    \hline
    Class & Communicator  & Matrix type & Interface to \\
    \hline
    \tt Amesos\_Lapack       & serial   & general & LAPACK \\
    \tt Amesos\_Klu          & serial   & general & KLU \\
    \tt Amesos\_Umfpack      & serial   & general & UMFPACK 4.3 \\
    \tt Amesos\_Pardiso      & serial/OMP   & general & PARDISO 1.2.3 \\
    \tt Amesos\_Taucs        & serial   & symmetric & TAUCS 2.2 \\
    \tt Amesos\_Superlu      & serial   & general & SuperLU 3.0 \\
    \tt Amesos\_Superludist & parallel & general & SuperLU\_DIST 2.0 \\
    \tt Amesos\_Mumps        & parallel & SPD, sym, general & MUMPS 4.5.4 \\
    \tt Amesos\_Dscpack      & parallel & symmetric & DSCPACK 1.0 \\
    \hline
  \end{tabular}
  \caption{Supported interfaces. ``serial'' means that the supported
    direct solver is serial. In this case, when solving with 
    more than one processor, the linear problem is gathered to process
    0, here solved, then the solution is broadcasted to the distributed
    solution 
    vector. ``serial/OMP'' means that the solver does not support MPI, but it
    can take advantage of OMP. Amesos will consider the solver as serial
    (therefore, the linear system is gathered on processor zero), but the user
    can still specify the number of OMP processes.
    ``parallel'' means that a subset or all the processes in the current
    communicator will be used by the solver. ``general'' means general
    unsymmetric 
    matrix. If ``sym'' (symmetric matrix) or  ``SPD'' (symmetric
    positive definite), the direct solver library can take advantage of
    that particular matrix property.}
  \label{tab:classes}
\end{table}

Each \amesos\ interface automatically selects the default
parameters defined by the supported solver. In most cases, these values
are a robust and reliable choice for most applications. If required, the user can
tune some of the parameters by using a parameter
list, which can be created with the following instructions:
\begin{verbatim}
Teuchos::ParameterList List;
\end{verbatim}
Parameters can be set using method \verb!set()!:
\begin{verbatim}
List.set(ParameterName,ParameterValue);
\end{verbatim}
\verb!ParameterName! is a string containing the parameter name,
and \verb!ParameterValue! is any valid C++ object that specifies the
parameter value (for instance, an integer, a pointer to an array or to
an object). The list of parameters that affect all \amesos\ solvers are
reported in Section~\ref{sec:parameters}, while parameters that are specific
to a given solver (if any) are reported in the Section of this document
dedicated to that solver. Once a list is created, parameters can be set using
method \verb!SetParameters(List)!.

\begin{remark}
All \amesos\ object are derived from pure virtual class 
\verb!Amesos_BaseSolver!. A pure virtual class is a class that
defines interfaces only, and contains no executable code. Pure virtual classes cannot
be instantiated; however, it is possible to declare and use pointers and
references to a pure virtual class, as normally done with class
{\tt Amesos\_BaseSolver}.
\end{remark}  

\begin{remark}
\amesos\ is an interface to other
  packages, mainly developed outside the Trilinos framework. In order to
  use those packages, the user should carefully check copyright and
  licensing of those third-party codes.  Please refer to the web page or
  the documentation of each particular package for details.
\end{remark}  
  
\begin{remark}
\amesos\ is used by other \trilinos\ packages. In particular, \ifpack\
  can take advantage of \amesos\ to define additive overlapping domain
  decomposition preconditioners (of Schwarz type), by using \amesos'
  factorizations to solve the local problems; see~\cite{ifpack-guide}.
Another package, \ml, takes
advantages of the \amesos\ interfaces to solve the coarse problem
that arises in multilevel preconditioners; see~\cite{ml-guide}.
\end{remark}

%-----------------------------------------------------------------------------
\subsection{Supported Matrix Formats}
\label{sec:matrix}
%-----------------------------------------------------------------------------

Table~\ref{tab:matrices} reports the supported matrix types for all the 
\amesos\ classes. In the table, ``Transp'' means that \amesos\ 
can solve both the linear system with the linear system matrix and with 
its transpose.  `$\bullet$' means that the interface can take advantage of the given matrix 
format, `--' means that it doesn't.

\begin{table}[tbhp]
  \centering
  \begin{tabular}{| p{3.5cm} | c | c | c | c | }
    \hline
    Class & Transp & Epetra\_RowMatrix & Epetra\_CrsMatrix & Epetra\_VbrMatrix \\
    \hline
    Amesos\_Lapack       & yes & $\bullet$ & $\bullet$ & -- \\
    Amesos\_Klu          & yes & $\bullet$ & $\bullet$ & -- \\
    Amesos\_Umfpack      & yes & $\bullet$ & $\bullet$ & -- \\
    Amesos\_Pardiso      & no  & $\bullet$ & -- & -- \\
    Amesos\_Taucs        & no  & $\bullet$ & -- & -- \\
    Amesos\_Superlu      & no  & $\bullet$ & $\bullet$ & -- \\
    Amesos\_Superludist  & no  & $\bullet$ & $\bullet$ & -- \\
    Amesos\_Mumps        & yes & $\bullet$ & -- & -- \\
    Amesos\_Dscpack      & yes & $\bullet$        & -- & -- \\
    \hline
  \end{tabular}
  \caption{Supported matrix formats. ``Transp'' means that \amesos\
    can solve both the linear system with the linear system matrix and with its
transpose.
`$\bullet$' means that the interface
    can take advantage of the given matrix format, `--' means that it doesn't.}
  \label{tab:matrices}
\end{table}

%-----------------------------------------------------------------------------
\subsection{Parameters for All \amesos\ Solvers}
\label{sec:parameters}
%-----------------------------------------------------------------------------

We now list all the parameters that may affect all the \amesos\ solvers.
To know whether a specific interface supports a given parameter, we
refer to table~\ref{tab:options}.

\vskip .1in \choicebox{\tt MatrixType}{Set it to {\tt SPD} if the matrix
  is symmetric positive definite, to {\tt symmetric} if symmetric, and
  to {\tt general} is the matrix is general unsymmetric.}

\choicebox{\tt Threshold}{In the conversion from Epetra\_RowMatrix to a
  solver's format, do not include elements whose absolute value is
  below the specified threshold.}

\choicebox{\tt AddZeroToDiag}{If {\tt true}, insert a zero element on the
  diagonal of the matrix (in the format required by the supported direct solver library) 
  for each row with no diagonal element.}

\choicebox{\tt PrintTiming}{Print some timing information when the
  \amesos\ object is destroyed.}

\choicebox{\tt PrintStatus}{Print some information about the linear
  system and the solver when the \amesos\ object is destroyed.}

\choicebox{\tt ComputeVectorNorms}{After solution, compute the 2-norm of
  each vector in the Epetra\_MultiVector $B$ and $X$. }

\choicebox{\tt ComputeTrueResidual}{After solution, compute the real
  residual $\| B - A X \|_2$ for all vectors in Epetra\_MultiVector. }

\choicebox{\tt MaxProcs}{If positive, the linear system matrix will be
  distributed on the specified number of processes only (or the all the
  processes in the MPI communicator if the specified number is greater).
  If {\tt MaxProcs=-1}, \amesos\ will estimate using internal heuristics
  the optimal number of processes that can efficiently solve the linear
  system. If {\tt MaxProcs=-2}, \amesos\ will use the square root of the
  number of processes.  If {\tt MaxProcs=-3}, all processes in the
  communicator will be used. \\ This option may require the conversion
  of a C++ MPI communicator to a FORTRAN MPI communicator. If this is
  not supported, the specified value of {\tt MaxProcs} will be ignored,
  and all the processes in the Epetra communicator will be used.}

\choicebox{\tt OutputLevel}{If {\tt 0}, no output is printed on the
  standard output. If {\tt 1}, output is reported as specified by other
  parameters. If {\tt 2}, all output is printed (this is equivalent to
  {\tt PrintTiming == true}, {\tt PrintStatus == true}, {\tt
    ComputeVectorNorms == true}, {\tt ComputeTrueResidual == true}).}

\choicebox{\tt Refactorize}{
  ``Refactorization'' of a matrix refers to the use of a prior
  symbolic and numeric factorization (including row and column
  ordering), to factorize a subsequent matrix using the same
  pivot ordering.  This can be significantly faster, but the
  numerical quality of the factorization may suffer.
  If {\tt true}, then attempt to re-use the existing
  symbolic and numeric factorization, to factorize a new matrix using
  the identical pivot ordering (both row and column ordering) as
  a prior pivot-capable factorization.}

\choicebox{\tt RcondThreshold}{After a refactorization,
    an estimate of the reciprocal of the condition number is computed.
    If this estimate is too small (less than {\tt RcondThreshold}),
    then the pivot-less factorization is aborted, and the matrix
    is factorized again with normal numerical pivoting.}

\choicebox{\tt ScaleMethod}{Most methods can scale the input matrix
prior to factorization.  This typically improves the quality of
the factorization and reduces fill-in as well.  Setting this
parameter to zero turns off scaling.  A value of 1 selects the
method's default scaling method (which may in fact be not to
scale at all).  A value of 2 means to scale the matrix
using the first non-default method the solver has, 3
means to use its 2nd alternative method, and so on.}

\smallskip

\begin{sidewaystable}[tbhp]
  \centering
  \begin{tabular}{| p{5cm} | p{2cm} | p{2cm} | c | c | c
      | c | c | c | c | c |}
    \hline
    option & type & default value & \rotatebox{90}{KLU} &
    \rotatebox{90}{UMFPACK} & 
    \rotatebox{90}{PARDISO} & 
    \rotatebox{90}{TAUCS} & 
    \rotatebox{90}{SuperLU\_DIST} &
    \rotatebox{90}{MUMPS} & \rotatebox{90}{LAPACK} &
    \rotatebox{90}{DSCPACK 1.0 } \\    
    \hline
    % -------------------------------------------------------------------
     \tt MatrixType         & \tt string & \tt general& --        & -- &  --
     &  --  & -- & $\bullet$ & -- & -- \\
    % -------------------------------------------------------------------
     \tt Threshold          & \tt double & \tt 0.0    & --        & --
     &  --         & --   &  -- & --& -- & --  \\
    % -------------------------------------------------------------------
     \tt AddZeroToDiag      & \tt bool   & \tt false  & --        & --
     & $\bullet$ & $\bullet$ & $\bullet$ & -- & -- & -- \\
    % -------------------------------------------------------------------
     \tt PrintTiming        & \tt bool   & \tt false  & $\bullet$ & $\bullet$
     & $\bullet$ & $\bullet$ & -- & $\bullet$ & $\bullet$ & $\bullet$ \\
    % -------------------------------------------------------------------
    \tt PrintStatus         & \tt bool   & \tt false  & $\bullet$ & $\bullet$
    & $\bullet$ & $\bullet$ & $\bullet$  & $\bullet$ & $\bullet$ & $\bullet$ \\
    % -------------------------------------------------------------------
    \tt MaxProcs            & \tt int    & -1         & --        & --
    & -- & -- &  $\bullet$  & $\bullet$  & $\bullet$ & $\bullet$  \\
    % -------------------------------------------------------------------
    \tt MaxProcsMatrix      & \tt int    & -4         & --        & --
    & -- & -- &  -- & $\bullet$  & -- &   --    \\
    % -------------------------------------------------------------------
    \tt ComputeVectorNorms  & \tt bool   & \tt false  & $\bullet$ & $\bullet$
    & $\bullet$ & $\bullet$ &  $\bullet$ & $\bullet$ & $\bullet$ & $\bullet$ \\
    % -------------------------------------------------------------------
    \tt ComputeTrueResidual & \tt bool   & \tt false  & $\bullet$ & $\bullet$
    & $\bullet$ & $\bullet$ &  $\bullet$ & $\bullet$ & $\bullet$  & $\bullet$\\
    % -------------------------------------------------------------------
    \tt OutputLevel         & \tt int    & \tt 1      & $\bullet$ & $\bullet$
    & $\bullet$ & $\bullet$ &  $\bullet$ & $\bullet$ & $\bullet$  & $\bullet$\\
    % -------------------------------------------------------------------
%---- added by Tim Davis:
    \tt Refactorize         & \tt bool   & \tt false  & $\bullet$ & -- & -- & -- &  -- & -- & -- & -- \\
    % -------------------------------------------------------------------
    \tt RcondThreshold      & \tt double & $10^{-12}$ & $\bullet$ & -- & -- & -- &  -- & -- & -- & -- \\
    % -------------------------------------------------------------------
    \tt ScaleMethod         & \tt int    & 1          & $\bullet$ & -- & -- & -- &  -- & -- & -- & -- \\
%----
    \hline
  \end{tabular}
  \caption{Supported options. `$\bullet$' means that the interface
    supports the options, `--' means that it doesn't.}
  \label{tab:options}
\end{sidewaystable}

Solver-specific parameters are reported in each direct solver's subsection.
The general procedure is to create a sublist with a given name (for
instance, the sublist for MUMPS is "mumps"), then set all the
solver's specific parameters in this sublist. 
For example, to enable MUMPS'
output, one can proceed as follows:
\begin{verbatim}
// defines here the entries of ictnl
Teuchos::ParameterList & MumpsList = AmesosList.sublist("mumps");
MumpsList.set("ICTNL(2)", 6);
\end{verbatim}
Parameters and sublists not recognized are simply ignored. Recall that
parameter names are case sensitive!

%-----------------------------------------------------------------------------
\section{Configuring and Installing \amesos}
\label{sec:configuration}
%-----------------------------------------------------------------------------

\amesos\ is distributed through the Trilinos project, and can be
downloaded from the web site
\verb!http://trilinos.sandia.gov/download!.

\amesos\ requires two other {\sc Trilinos} packages, {\sc Epetra} and {\sc
  Teuchos}. 
Each of the \amesos\ classes provides an interface to a third-party direct
sparse solver code (exception to this rule is KLU, which is
  distributed within \amesos). In order to configure and compile a given
interface, the user must first install the underlying direct sparse
solver code. Generally, the BLAS library is required. Some solvers may
need CBLACS, LAPACK, BLACS, ScaLAPACK. 

\amesos\ is configured and built using the GNU autoconf~\cite{Autoconf}
and automake~\cite{Automake} tools. To configure \amesos\ from the
Trilinos top directory, a possible procedure is as follows.  Let
\verb!$TRILINOS_HOME! be a shell variable representing the location
of the Trilinos source directory, and \verb!%! the shell prompt sign.  
Let us suppose that we want to configure \amesos\ on a
LINUX machine with MPI, with support
for KLU and UMFPACK. Header files for UMFPACK are located
in directory \verb!/usr/local/umfpack/include!, while the library,
called \verb!libumfpack.a! is located in \verb!/usr/local/umfpack/lib!.
The configure like will look like:
\begin{verbatim}
% cd $TRILINOS_HOME
% mkdir LINUX_MPI
% cd LINUX_MPI
% ../configure \
  --enable-mpi \
  --prefix=$TRILINOS_HOME/LINUX_MPI \
  --enable-amesos \
  --enable-amesos-klu \
  --enable-amesos-umfpack \
  --with-incdirs="-I/usr/local/umfpack/include" \
  --with-ldflags="-L/usr/local/umfpack/lib" \
  --with-libs="-lumfpack"
% make
% make install
\end{verbatim}
Other flags may be required depending on the location of
 MPI, BLAS and LAPACK. Supported architectures are reported 
 in Table~\ref{tab:arch}.

\begin{remark}
The KLU sources are distributed with the \amesos\ package. We strongly
encourage to configure \amesos\ with KLU support. KLU and LAPACK are the only
interface that are turned on by default.
\end{remark}

\begin{sidewaystable}[tbhp]
  \centering
  \begin{tabular}{| l  l | c | c | c | c | c | c| c|}
    \hline
    Architecture & Communicator & LAPACK  & KLU       & UMFPACK   & SuperLU &  SuperLU\_DIST 2.0 & MUMPS 4.3.1 & DSCPACK 1.0\\
    \hline
    LINUX           & SERIAL  & $\bullet$ & $\bullet$ & $\bullet$ & $\bullet$ & --        & --        & --        \\
    % --------------------------------------------------------
    LINUX, GNU      & LAM/MPI & $\bullet$ & $\bullet$ & $\bullet$ & $\bullet$ & $\bullet$ & --        & $\bullet$ \\
    % --------------------------------------------------------
    LINUX, Intel    & MPICH   & $\bullet$ & $\bullet$ & $\bullet$ & --        & --        & $\bullet$ & $\bullet$ \\
    % --------------------------------------------------------
    SGI 64          & MPI     & $\bullet$ & $\bullet$ & $\bullet$ & --        & $\bullet$ & $\bullet$ & --        \\
    % --------------------------------------------------------
    DEC/Alpha       & MPI     & $\bullet$ & $\bullet$ & $\bullet$ & --        & --        & --        & --        \\
    % --------------------------------------------------------
    MAC OS X/G4     & MPICH   & $\bullet$ & $\bullet$   & --      & --        & --        & --        & --        \\
    % --------------------------------------------------------
    Sandia Cplant   & MPI     & $\bullet$ & $\bullet$ & $\bullet$ & --        & $\bullet$ & $\bullet$ & --        \\
    % --------------------------------------------------------
    Sandia ASCI Red & MPI     & $\bullet$ & $\bullet$ & $\bullet$ & --        & $\bullet$ & --        & --        \\
    \hline
  \end{tabular}
  \caption{Supported architectures for various interfaces. 
  `$\bullet$' means that the interface has been successfully compiled, 
  `--' means that it has not been tested.}
  \label{tab:arch}
\end{sidewaystable}

\smallskip

Up-to date documentation for \amesos\ is maintained through Doxygen, and it
can be generated with the following commands:
\begin{verbatim}
% cd $TRILINOS_HOME/packages/amesos
% cd doc
% doxygen
% <your-browser> html/index.html
\end{verbatim}

%-----------------------------------------------------------------------------
\section{Supported Solvers}
\label{sec:classes}
%-----------------------------------------------------------------------------

This Section details the solvers supported by \amesos.
The LAPACK interface is presented in Section~\ref{sec:lapack}, the KLU
interface in Section~\ref{sec:klu}, the UMFPACK interface
in Section~\ref{sec:umfpack},
  the PARDISO interface in Section~\ref{sec:pardiso},
  the TAUCS interface in Section~\ref{sec:taucs},
  the SuperLU interface in Section~\ref{sec:superlu}, the interface to
  SuperLU\_DIST in Section~\ref{sec:superludist}, the MUMPS interface in
  Section~\ref{sec:mumps}, and finally the DSCPACK interface in
  Section~\ref{sec:dscpack}. 

%-----------------------------------------------------------------------------
\subsection{Interface to LAPACK}
\label{sec:lapack}
%-----------------------------------------------------------------------------

\amesos\ must be configured with the option
\verb!--enable-amesos-lapack! in order to use the LAPACK interface.
Header files and the LAPACK library are automatically located by {\tt
  configure}.

LAPACK is a (suite of) serial solver(s). \amesos\ will gather all matrix rows
on processor zero before the symbolic factorization, and all matrix values
before the numeric factorization. On process 0, the matrix will be converted
to dense storage, using \verb!Epetra_SerialDenseMatrix! objects.  A call to
\verb!Solve()! requires a gather of the right-hand side on process 0, the
local solution of the linear system, and finally a scatter operation, to
redistribute as necessary the solution vector.

%-----------------------------------------------------------------------------
\subsection{Interface to KLU}
\label{sec:klu}
%-----------------------------------------------------------------------------

KLU is Timothy A.~Davis' implementation of Gilbert-Peierl's left-looking sparse
partial pivoting algorithm, with Eisenstat and Liu's symmetric pruning.
It doesn't exploit dense matrix kernels, but it is the only sparse LU
factorization algorithm known to be asymptotically optimal, in the sense
that it takes time proportional to the number of floating-point
operations. It is the precursor to SuperLU, thus the name ("Clark Kent
LU"). For very sparse matrices that do not suffer much fill-in (such as
most circuit matrices when permuted properly) dense matrix kernels do
not help, and the asymptotic run-time is of practical importance.

In order to use KLU, \amesos\ must be configured with the option
\verb!--enable-amesos-klu!.

%-----------------------------------------------------------------------------
\subsection{Interface to UMFPACK 4.3}
\label{sec:umfpack}
%-----------------------------------------------------------------------------

UMFPACK is a C package copyrighted by Timothy A.~Davis. More information
can be obtained at the web page
\verb!http://www.cise.ufl.edu/research/sparse/umfpack!.

\amesos\ must be configured with the option
\verb!--enable-amesos-umfpack! to use the UMFPACK interface.  The location of the header files should be
specified using \verb!--with-incdirs!, the location of the library with
\verb!--with-ldflags!, and the library to be linked by \verb!--with-libs!. See
Section~\ref{sec:configuration} for an example.

%-----------------------------------------------------------------------------
\subsection{Interface to PARDISO 1.2.3}
\label{sec:pardiso}
%-----------------------------------------------------------------------------

PARDISO is package to solve large sparse symmetric and non-symmetric linear
systems on shared memory multi-processors, developed at the Computer Science
Department of the University of Basel. A discussion of the algorithms used
in PARDISO and more information on the solver can be found at {\tt
  http://www.computational.unibas.ch/cs/scicomp} and in
  documents~\cite{oskl:04-etna,sg:04-fgcs}.

\amesos\ must be configured with the option
\verb!--enable-amesos-pardiso! to use the PARDISO interface.  
The location of the header files should be
specified using \verb!--with-incdirs!, the location of the library with
\verb!--with-ldflags!, and the library to be linked by \verb!--with-libs!. See
Section~\ref{sec:configuration} for an example.

The Amesos interface to PARDISO supports non-symmetric matrices only 
(from the PARDISO manual, the matrix type is 11). PARDISO control parameters
are specified using the IPARM vector. The user can 
  
\medskip

The PARDISO interface will look for a sublist, called
\verb!Pardiso!. The user is referred to PARDISO
manual for a detailed explanation of the reported parameters. 

\vskip .1in

\choicebox{\tt IPARM(1)}{({\tt int}) Use default values.
  Default: {\tt 0}.}

\choicebox{\tt IPARM(2)}{({\tt int}) Fill-in reduction reordering.
  Default: {\tt 0}.}

\choicebox{\tt IPARM(3)}{({\tt int}) Number of processors.
  Default: {\tt 1}.}

\choicebox{\tt IPARM(4)}{({\tt int}) Preconditioned CGS.
  Default: {\tt 0}.}

\choicebox{\tt IPARM(8)}{({\tt int}) 
  Default: {\tt 0}.}

\choicebox{\tt IPARM(10)}{({\tt int}) Iterative refinement steps.
  Default: {\tt 0}.}

\choicebox{\tt IPARM(11)}{({\tt int}) MPS scaling of the unsymmetric
  reoredering.
  Default: {\tt 8}.}

\choicebox{\tt IPARM(18)}{({\tt int}) Number of nonzeros in factors.
  Default: {\tt -1}.}

\choicebox{\tt IPARM(19)}{({\tt int}) MFlops in factorization.
  Default: {\tt 0}.}

\choicebox{\tt IPARM(21)}{({\tt int}) Pivoting for indefinite symmetric
  matrices.
  Default: {\tt 1}.}

%-----------------------------------------------------------------------------
\subsection{Interface to TAUCS 2.2}
\label{sec:taucs}
%-----------------------------------------------------------------------------

TAUCS, authored by S. Toledo, is a serial Cholesky
solver~\cite{rozin04locality, rotkin04design,irony04parallel}. 
\amesos\ must be configured with the option
\verb!--enable-amesos-taucs! to use the TAUCS interface.  
The location of the header files should be
specified using \verb!--with-incdirs!, the location of the library with
\verb!--with-ldflags!, and the library to be linked by \verb!--with-libs!. See
Section~\ref{sec:configuration} for an example.

%-----------------------------------------------------------------------------
\subsection{Interface to SuperLU 3.0}
\label{sec:superlu}
%-----------------------------------------------------------------------------

SuperLU, written by Xiaoye S. Li, is a serial solver
written in ANSI C. It is copyrighted by The Regents of the University of
California, through Lawrence Berkeley National Laboratory.  We refer to
the web site
\verb!http://www.nersc.gov/~xiaoye/SuperLU!
and to the SuperLU manual~\cite{superlu-manual} for more
information.

In order to interface with SuperLU\_DIST 2.0, \amesos\ must be configured with
the option \verb!--enable-amesos-superlu!.
The location of the header files should be specified using \verb!--with-incdirs!,
the location of the library with \verb!--with-ldflags!, and the library
to be linked by \verb!--with-libs!. See Section~\ref{sec:configuration} for an
example.

%-----------------------------------------------------------------------------
\subsection{Interface to SuperLU\_DIST 2.0}
\label{sec:superludist}
%-----------------------------------------------------------------------------

SuperLU\_DIST, written by Xiaoye S. Li, is a parallel extension to the
serial SuperLU library.  SuperLU\_DIST is written in ANSI C, using MPI
for communication, and it is targeted for the distributed memory
parallel machines.  SuperLU\_DIST includes routines to handle both real
and complex matrices in double precision. However, as \amesos\ is
currently based on the Epetra package (that does not handle complex
matrices), only double precision matrices can be considered.

Amesos\_Superludist can solve the linear system on a subset of the
processes, as specified in the parameters list. This is done by
creating a new process group derived from the MPI group of the
Epetra\_Comm object, with function \verb!superlu_gridinit()!. 

In order to interface with SuperLU\_DIST 2.0, \amesos\ must be configured with
the option \verb!--enable-amesos-superludist!.
The location of the header files should be specified using \verb!--with-incdirs!,
the location of the library with \verb!--with-ldflags!, and the library
to be linked by \verb!--with-libs!. See Section~\ref{sec:configuration} for an
example.

\medskip

The SuperLU\_DIST constructor will look for a sublist, called
\verb!Superludist!.  The following parameters reflect the behavior of
SuperLU\_DIST options argument, as specified in the SuperLU\_DIST
manual~\cite[pages 55--56]{superlu-manual}. The user is referred to this
manual for a detailed explanation of the reported parameters. Default
values are as reported in the SuperLU\_DIST manual.

\vskip .1in

\choicebox{\tt Fact}{({\tt string}) Specifies whether or not the factored form
  of the matrix $A$ is supplied on entry and, if not, how the matrix will
  be factored. It can be: {\tt DOFACT, SamePattern,
    SamePattern\_SameRowPerm, FACTORED}. Default: {\tt
    SamePattern\_SameRowPerm}.}

\choicebox{\tt Equil}{({\tt bool}) Specifies whether to equilibrate the system of
  not. Default: {\tt true}.}

\choicebox{\tt ColPerm}{({\tt string}) Specifies the column ordering
  strategy.  It can be: {\tt NATURAL, MMD\_AT\_PLUS\_A, MMD\_ATA,
    COLAMD, MY\_PERMC}.  Default: {\tt MMD\_AT\_PLUS\_A}.}

\choicebox{\tt perm\_c}{({\tt int *}) Specifies the ordering to use when {\tt
    ColPerm = MY\_PERMC}.}

\choicebox{\tt RowPerm}{({\tt string}) Specifies the row ordering
  strategy.  It can be: {\tt NATURAL, LargeDiag, MY\_PERMR}.  Default:
  {\tt LargeDiag}.}

\choicebox{\tt perm\_r}{({\tt int *}) Specifies the ordering to use when {\tt
    RowPerm = MY\_PERMR}.}

\choicebox{\tt ReplaceTinyPivot}{({\tt bool}) Specifies whether to replace the
  tiny diagonals with $\varepsilon \| A \|$ during LU
  factorization. Default: {\tt true}.}

\choicebox{\tt IterRefine}{({\tt string}) Specifies how to perform iterative
  refinement. It can be: {\tt NO, DOUBLE, EXTRA}. Default: {\tt
    DOUBLE}.}

%-----------------------------------------------------------------------------
\subsection{Interface to MUMPS 4.5.4}
\label{sec:mumps}
%-----------------------------------------------------------------------------

MUMPS (``MUltifrontal Massively Parallel Solver'') is a parallel direct
solver, written in FORTRAN 90 with a C interface, copyrighted by P. R.
Amestoy, I. S.  Duff, J. Koster, J.-Y.  L'Excellent. Up-to-date copies
of the MUMPS package can be obtained from the Web page
\begin{verbatim}
http://www.enseeiht.fr/apo/MUMPS/
\end{verbatim}

MUMPS can solve the original system~(\ref{eq:linear_system}), as well as
the transposed system, given an assembled or elemental matrix. Note that
only the assembled format is supported by Amesos\_Mumps. Mumps offers,
among other features, error analysis, iterative refinement, scaling of
the original matrix, computation of the Schur complement with respect to a prescribed
subset of rows. Reordering techniques can take advantage of PORD
(distributed within MUMPS), or METIS~\cite{METIS}\footnote{At this time,
  METIS ordering is not supported by class {\tt Amesos\_Mumps}.}. 
For details about the algorithms and the
implementation, as well as of the input parameters, we refer
to~\cite{mumps-manual}

In order to interface with MUMPS 4.5.4, \amesos\ must be configured with the
option \verb!--enable-amesos-mumps!. The MUMPS interface can be compiled in
serial or in MPI mode. In this latter case, the interface can take be used on
a subset of the processes. To that aim, it must be possible to convert from a
C++ MPI communicator to a FORTRAN MPI communicator. Such a conversion is not
always possible. In you experience compilation problems with {\tt
Amesos\_Mumps}, you can try the option {\tt
--disable-amesos-mumps\_mpi\_c2f}.

The location of the header files should be specified using
\verb!--with-incdirs!, the location of the library with \verb!--with-ldflags!,
  and the library to be linked by \verb!--with-libs!. See
  Section~\ref{sec:configuration} for an example.

\medskip

The MUMPS constructor will look for a sublist, called
\verb!mumps!. 
All ICNTL($i$), $i=1,\ldots,40$ and CNTL(j), $j=1,\ldots,5$ options can be set
by
The user can set all the MUMPS's parameters by se



integer array ICNTL and the double array CNTL to the parameters list,
or by using the functions reported at the end of this section.

\vskip .1in
\choicebox{\tt ICTNL($i$)}{({\tt int}) This allows to set all ICNTL($i$),
  $i=1,\ldots,40$ parameters. For example, {\tt list.set("ICNTL(2)", 6)} sets
    the second value of the ICNTL array. Numbering are as reported on the
    MUMPS manual, starting from 1.}

\choicebox{\tt CTNL($i$)}{({\tt double}) This allows to set all CNTL($i$),
  $i=1,\ldots,5$ parameters. Numbering are as reported on the
    MUMPS manual, starting from 1.}

\choicebox{\tt PermIn}{({\tt int *}) Use integer vectors of size
  NumGlobalElements (global dimension of the matrix) as given ordering.
  {\tt PermIn} must be defined on the host only, and allocated by the
  user, if the user sets ICNTL(7) = 1.}
  
%%\choicebox{\tt Maxis}{({\tt int}) Set Maxis value.}

%%\choicebox{\tt Maxs}{({\tt int}) Set Maxs value.}

\choicebox{\tt ColPrecScaling}{({\tt double *}) Use double precision vectors
  of size NumGlobalElements (global dimension of the matrix) as scaling
  for columns and rows. The double vector must be defined on the host
  only, and allocated by the user, if the user sets ICNTL(8) = -1.}

\choicebox{\tt RowPrecScaling}{({\tt double *}) Use double precision vectors
  of size NumGlobalElements (global dimension of the matrix) as scaling
  for columns and rows. The double vector must be defined on the host
  only, and allocated by the user, if the user sets ICNTL(8) = -1.}

\smallskip

Other functions are available to check the output values. The following
Amesos\_Mumps methods are {\sl not} supported by the Amesos\_BaseSolver
class; hence, the user must create an Amesos\_Mumps object in order to
take advantage of them.

\medskip

\choicebox{\tt double* GetRINFO()}{
Gets the pointer to the RINFO array (defined on all processes).
}

\choicebox{\tt int* GetINFO()}{
Gets the pointer to the INFO array (defined on all processes).
}

\choicebox{\tt double* GetRINFOG()}{
Gets the pointer to the RINFOG array (defined on processor 0 only).
}

\choicebox{\tt int* GetINFOG() }{
Gets the pointer to the INFOG array (defined on processor 0 only).
}

A functionality that is peculiar to MUMPS is the ability to return
the Schur complement matrix, with respect to a specified set of nodes.

\begin{verbatim}
int ComputeSchurComplement(bool flag, int NumSchurComplementRows, 
                           int* SchurComplementRows);
\end{verbatim}
This method computes (if \verb!flag! is true) the Schur complement with
respect to the set of indices included in the integer array {\tt
  SchurComplementRows} , of size {\tt NumSchurComplementRows}. This is a
{\sl global} Schur complement, and it is formed (as a dense matrix) on
processor 0 only. Method
\begin{verbatim}
Epetra_CrsMatrix* GetCrsSchurComplement()
\end{verbatim}
returns the Schur complement in an Epetra\_CrsMatrix, on processor 0
only. No checks are performed to see whether this action is legal or not
(that is, if the call comes after the solver has been invoked).
The returned Epetra\_CrsMatrix must be free'd by the user. Method
\begin{verbatim}
Epetra_SerialDenseMatrix * GetDenseSchurComplement();
\end{verbatim}
returns the Schur complement as a Epetra\_SerialDenseMatrix
(on processor 0 only), to be free'd by the user.

\smallskip

As an example, the following fragment of code shows how to use MUMPS to
obtain the Schur complement matrix with respect to a given subsets of
nodes.  First, we need to create an parameter list, and an Amesos\_Mumps
object.
\begin{verbatim}
Teuchos:::ParameterList params;
Amesos_Mumps * Solver;
Solver = new Amesos_Mumps(*Problem,params);
\end{verbatim}
Then, we define the set of nodes that will constitute the Schur
complement matrix. This must be defined on processor 0 only. For
instance, one may have:
\begin{verbatim}
int NumSchurComplementRows = 0;
int* SchurComplementRows = NULL;
if (Comm.MyPID() == 0) 
{
  NumSchurComplementRows = 4;
  SchurComplementRows = new int[NumSchurComplementRows];
  SchurComplementRows[0] = 0;
  SchurComplementRows[1] = 1; 
  SchurComplementRows[2] = 2;
  SchurComplementRows[3] = 3;
}
\end{verbatim}
Now, we can ask for the Schur complement using
\begin{verbatim}  
Solver->ComputeSchurComplement(true, NumSchurComplementRows,
                               SchurComplementRows);
\end{verbatim}
The Schur complement matrix can be obtain after the solver phase:  
\begin{verbatim}  
Solver->Solve();
Epetra_CrsMatrix * SC;
SC = Solver->GetCrsSchurComplement();
Epetra_SerialDenseMatrix * SC_Dense;
SC_Dense = Solver->GetDenseSchurComplement();
\end{verbatim}

%-----------------------------------------------------------------------------
\subsection{Interface to DSCPACK 1.0}
\label{sec:dscpack}
%-----------------------------------------------------------------------------

DSCPACK, written by  Padma Raghavan, is a domain-separator code for the
parallel solution of sparse linear system.
DSCPACK provides a variety of sparsity preserving (fill-reducing) ordering and
computes either an $LL^T$ (Cholesky) or $LDL^T$ factorization of the linear
system matrix. This solver is written in C, and it uses MPI for
inter-processor communication, and the BLAS library for improved
chace-performances. The implementation is based on the idea of partitioning
the sparse matrix into domains and separators. 

We refer to the web site
\verb!http://www.cse.psu.edu/~ragavan/dscpack!
and to the DSCPACK manual~\cite{dscpack-manual} for more
information.

\amesos\ must be configured with the option
\verb!--enable-amesos-dscpack! to use DSCPACK.
The location of the header files should be specified using \verb!--with-incdirs!,
the location of the library with \verb!--with-ldflags!, and the library
to be linked by \verb!--with-libs!. See Section~\ref{sec:configuration} for an
example.

DSCPACK solves the linear system using a number of processors that is a power
of 2. If necessary, we linear system matrix will be automatically
redistributed on the highest number of processors (either all the
processors, or the number specified in \verb!MaxProcs!) that is a power of 2.

%-----------------------------------------------------------------------------
\section{Guide to the Examples}
\label{sec:examples}
%-----------------------------------------------------------------------------

The \amesos\ distribution contains examples in subdirectory
\begin{verbatim}
$TRILINOS_HOME/packages/amesos/example
\end{verbatim}
Most of the example requires \amesos\ to be configured with support for
\triutils. \triutils\ is a Trilinos package, automatically compiled unless
the user specifies
\begin{verbatim}
--disable-triutils
\end{verbatim}
or
\begin{verbatim}
--disable-default-packages
\end{verbatim}
\triutils\ is used to generate the linear system matrix. New users can
start from file
\begin{verbatim}
$TRILINOS_HOME/packages/amesos/example/example_AmesosFactory.cpp
\end{verbatim}
which contains detailed comments about all the \amesos\ commands.
Example
\begin{verbatim}
$TRILINOS_HOME/packages/amesos/example/example_AmesosFactory_HB.cpp
\end{verbatim}
shows how to read a matrix stored in Harwell/Boeing format, redistribute it
to all the processes used in the computation, and use \amesos\ to solve the
corresponding linear system. Finally, example
\begin{verbatim}
$TRILINOS_HOME/packages/amesos/example/example_AmesosFactory_Tridiag.cpp
\end{verbatim}
creates a simple tridiagonal matrix, and solves the corresponding linear
system.

\bibliographystyle{plain}
\bibliography{AmesosBibliography}

\end{document}
