Development Tools

 
 
 
 
 
 

The partner computing centers in ScalaLife provide the necessary tools for software development, along with the core simulation packages. The tools include version control systems, build environemnts, compilers, debuggers, performance tools and special libraries. Analysis and visualization software is also provided.

Some of the provided tools require special or paid licenses. Users will need to obtain those licenses themselves, except for the cases where the centres provide system wide licensing scheme. Please contact support for more information.

Access to the tools (and pre-compiled applications)

Some of the centers use module systems to manage access to the tools. See below for details.

PDC at the Royal Institute of Technology (KTH)

Precompiled versions of the tools are provided through the Tcl modules systems on all clusters at PDC. Information about their usage through the module system can be found on the PDC website.

LRZ

LRZ uses the modules approach to manage the user environment for different software versions. This is the standard method available for user environment management on all Linux-based LRZ HPC platforms. More information can be found on the LRZ website.

BSC

BSC uses the module tools to manage the user environment to support  different software stack versions. More information about software and tools can be found at BSC website

OeRC

Tools such as the GNU build system and compilers which are part of the standard Linux distributions are available by default. The other tools can be added as modules. More information is available on the OeRC website.

Documentation

Links to documentation about the usage of the tools can be found in the performance section.

Provision of tools

The availability of the tools on the different platforms is presented in the table below. More details are available in the support tools matrix.

 

Tools Description Availability
Analysis tools
Gromacs analysis tools a large collecion of tools for analysis of molecular structures and trjectories. Comes bundled with the Gromacs package KTH: Povel, Lindgren

LRZ: all
BSC: MareNostrum, bscsmp
OeRC: Hal, Sal, Skynet, RedQueen, Queeg
R free software environment for statistical computing and graphics KTH: Povel, Lindgren

LRZ: all

BSC: MareNostrum, bscsmp
OeRC: Hal, Sal, RedQueen, Queeg
Build environments
Autotools GNU build system tools KTH: all

LRZ: all

BSC: all

OeRC: all
CMake cross-platform build system KTH: all

LRZ: all

BSC: all

OeRC: all
Compilers

GNU Compiler Collection
(g++/gcc/gfortran)

Integrated distribution of compilers for several programming languages. KTH: all

LRZ: all

BSC: all

OeRC: all (4.1.0)
IBM Compilers
(XL c/c++, fortran)
Compilers optimized for the IBM POWER systems BSC: MareNostrum, MariCel, tamariu
Intel Compilers
(icc/ifort)
Highly optimized compilers for Intel and Intel-compatible processors. KTH: all

LRZ: all

BSC: bscsmp, GPU clusters

OeRC: all (11.1)
Debuggers
DDD GNU DDD is a graphical front-end for command-line debuggers such as GDB LRZ: all

BSC: MareNostrum

OeRC: RedQueen, Queeg
GDB GNU Project debugger KTH: all

LRZ: all

BSC: all

OeRC: all
Intel Debugger
(IDB)
Part of the Intel suite of programming tools. It can be used also in a GDB interface mode. KTH: all

LRZ: all

BSC: bscsmp

OeRC: Hal, Sal, Skynet, RedQueen, Queeg (11.1)
Totalview powerful parallel debugger with multi-process/multi-threading support KTH: all

LRZ: all

BSC: MareNostrum, MariCel, bscsmp

OeRC: none
DDT  powerful parallel debugger with multi-process/multi-threading support KTH: all

LRZ: all
BSC: MareNostrum
OeRC: all
Valgrind memory debuggin and profiling framework KTH: all

LRZ: all

BSC: all

OeRC: Hal, Sal, Skynet, RedQueen, Queeg
Performance tools
Paraver flexible performance visualization and analysis tool KTH: all

LRZ: none

BSC: all

OeRC: none
Scalasca software tool that supports the performance optimization of parallel programs by measuring and analyzing their runtime behavior KTH: Povel

LRZ: all

BSC: MareNostrum

OeRC: none
Tau portable profiling and tracing toolkit for performance analysis of parallel programs written in Fortran, C, C++, Java, Python KTH: Povel

LRZ: all

BSC: MareNostrum

OeRC: Hal, Sal
VTune performance analyzer from Intel KTH: none

LRZ: SGI Altix 4700

BSC: none

OeRC: none
Vampir analysis framework for performance optimization KTH: none

LRZ: all

BSC: MareNostrum

OeRC: none
gprof GNU profiler KTH: all

LRZ: all

BSC: all

OeRC: all
Programming models and libraries
COMPSs Programming model for automatic parallelization KTH: none

LRZ: none

BSC: MareNostrum

OeRC: none
FFTW library for calculating discrete Fourier transforms KTH: all

LRZ: all

BSC: MareNostrum, bscsmb, taimaru, GPU cluster

OeRC: all (2.1.5, 3.1.2)
MKL Optimized math libraries from Intel KTH: all

LRZ: all

BSC: bscsmp

OeRC: all (11.1)
MPI message-passing interface for a distribute programming model KTH: all

LRZ: all
BSC: all

OeRC: all
CUDA/OpenCL compilers, libraries and development tools for accelerated calculations on NVIDIA GPGPUs KTH: all

LRZ: GVS cluster

BSC: GPU cluster

OeRC: Skynet
Version Control Systems
Git powerful  distributed control system KTH: all

LRZ: all

BSC: none

OeRC: all
SVN modern version of the classic CVS system KTH: Povel

LRZ: all

BSC: MareNostrum, bscsmp

OeRC: all
Visualization tools
Maestro unified interface for all modecular modeling software from Schrödinger KTH: none

LRZ: gvs

BSC: none

OeRC: none
Molden package for displaying Molecular Density KTH: all

LRZ: hlrb2

BSC: MareNostrum

OeRC: Hal, Sal
PyMol molecular visualization system KTH: all

LRZ: gvs

BSC: MareNostrum, bscsmp

OeRC: none
VMD molecular visualization program for displaying, animating, and analyzing KTH: none

LRZ: gvs

BSC: MareNostrum, bscsmp
OeRC: none

Analysis tools

Gromacs analysis tools

The Gromacs installation package comes with a large selection of over 60 analysis tools. They are very versatile and can be used to obtain detailed description of the dynamics of the simulated systems, and also of various properties of proteins, lipids, liquids and ions. The tools come bundled with the official Gromacs distribution.

Online documentation
Gromacs manual (pdf)

R

R is a free software environment for statistical computing and graphics.The R language has become a de facto standard among statisticians for the development of statistical software and is widely used for statistical software development and data analysis. R is part of the GNU project.


Manuals
Books related to R
 

Build Environements

Autotools

Autotools is a collection of programming tools which are the basis of the GNU build system. They include autoconf, automake, libtool and several other related programs. They are used to build the Gromacs and Dalton packages.

Autoconf project page and documentation
Automake project page and documentation
Libtool project page and documentation

CMake

CMake is an extensible, open-source system that manages the build process in an operating system and in a compiler-independent manner. It is being used by large software projects such as KDE.

The CMake project page
The CMake Documentation page
The CMake Wiki
The CMake FAQ
The CMake mailing list - very active and useful resource of help

Compilers

CUDA/OpenCL compilers

CUDA (Compute Unified Device Architecture) is developed by NVIDIA to allow programmers to access the power of the massively parallel streaming archicture of the GPU hardware. Compilers, debuggers, libraries, performance tools and code samples for Windows, Linux and MacOS platforms are provided on the project's website, along with extensive documentation for developers.

NVIDIA's CUDA zone
GPU computing portal (NVIDIA)
GPUcomputing.net research and development community

OpenCL (Open Computing Language) is an open-standard framework for developing and execution of parallel code on vendor-agnostic heterogenous systems.

Official website
OpenCL for NVIDIA
OpenCL for AMD

OpenCL for IBM Cell broadband Engine
gDEBugger CL - OpenCL Debugger, Profiler and Memory Analyzer

Online seminars

GPU Computing seminars (NVIDIA)

GNU compilers

GCC is an integrated distribution of compilers for several major programming languages. These languages currently include C, C++, Objective-C, Objective-C++, Java, Fortran, and Ada. They are developed as part of the GNU project.

For the Fortran 95 compiler, please also refer to the G95 development home page.

Documentation for users

Documentation for developers

IBM compilers

 

 

The IBM C/C++  and Fortran compliers offer advanced optimization technologies for creating highly-tuned code for POWER architectures.

XL C/C++ compilers

XL Fortran compiler

Optimized math libraries

IBM Linux compilers optimization center - Installation, getting started, programming and optimization guides from IBM. Includes compiler and language references as well as information about BlueGene systems.

Intel Compilers

Intel provides higly optimized compilers and libraries for Intel-compatible architectures. The compilers support also multi-threading, automatic vectorization, and OpenMP. The collection of tools comes with optimized libraries and powerful debugging tools.

Official site of the tools
The compilers are available free of charge for non-commercial use.
Intel compilers optimization guide
Optimization manuals for x86 and x86-64 processors

Debuggers

Alinea DDT

DDT (Distributed Debugging Tool) is a commercial parallel debugging tool for OpenMP multi-threaded and MPI distributed applications, which supports C, C++ and Fortran.

Official site

DDD

GNU DDD is a graphical front-end for command-line debuggers such as GDB, DBX, WDB, Ladebug, JDB, XDB, the Perl debugger, the bash debugger bashdb, the GNU Make debugger remake, or the Python debugger pydb.


DDD Manual

GDB

GDB (GNU debugger) is the standard debugging tool for the GNU Project software and supports several languages such as C, C++ and Fortran.

Official site
Documentation
Internals (for developers)
Tutorial on using GDB

Intel debugger (idb)

The Intel Debugger (IDB)  provides support for debugging programs written in C, C++, and Fortran It provides a choice of command-line and graphical user interface (GUI) on the Linux Eclipse platform. On Windows, Intel provides extension to the Microsoft Debugger that enables additional capabilities for debugging parallel programs.

Totalview

Totalview is a very powerful source code analysis tool with multi-process/multi-threading support. It supports C/C++ and Fortran sources and runs on Linux, MacOS and various UNIX based platforms.

The Totalview debugger, in the words of it's User's Guide, is "part of a suite of software development tools for debugging, analyzing and tuning the performance of programs, including multiprocess (MPI) and/or multithreading (OpenMP) programs".

 
The memory debugging component is also separately available as an easy to use tool: Memoryscape. The latter tool allows identification of memory leaks as well as memory profiling.
 
Finally, on x86_64 based systems, the Totalview Replay Engine is available, which allows you to roll back the state of your program to any point in the past.

Official site
Documentation
Video tutorials
Another Totalview tutorial

Valgrind

Valgrind is a framework that provides a suite of tools for memory debugging, management and profiling with multi-threading support. The suite includes six production-quality tools: a memory error detector, two thread error detectors, a cache and branch-prediction profiler, a call-graph generating cache and branch-prediction profiler, and a heap profiler.

It is a flexible system for debugging and profiling Linux executables for x86_64 architectures. The functionality of Valgrind covers

  • Memcheck/Addrcheck: Detection of memory-management problems

  • Cachegrind: Cache profiler. Detailed simulation of the I1, D1 and L2 caches is provided to pinpoint the sources of cache misses.

  • Callgrind: adds call graph tracing to cachegrind. It can be used to get call counts and inclusive cost for each call happening in your program. In addition to cachegrind, callgrind can annotate threads separately, and every instruction of disassembler output of your program with the number of instructions executed and cache misses incurred.

  • Massif: Memory consumption profiling.

  • Helgrind: Identify data races in multithreaded programs

Official site
User manual
Tools from the suite
Graphical interfaces

Performance Tools

PAPI

PAPI

PAPI aims to provide the tool designer and application engineer with a consistent interface and methodology for use of the performance counter hardware found in most major microprocessors. PAPI enables software engineers to see, in near real time, the relation between software performance and processor events.

Paraver

Paraver is part of CEPBA-Tools, an open source project developed by BSC. Paraver is a very powerful and flexible performance visualization and analysis tool based on traces that can be used to analyse any information that is expressed on its input trace format. The other main modules of the toolkit include Dimemas, a simulation tool for the parametric analysis of the behaviour of message-passing applications on a configurable parallel platform and Extrae that groups the instrumentation libraries to generate traces of different programming models that include MPI, OpenMP, hybrid, Pthreads...

Tutorial guidelines and presentations can be found on the documentation page

The tools can be downloaded from here

Periscope

Periscope

Periscope is a scalable automatic performance analysis tool currently under development at Technische Universitaet Muenchen. It consists of a frontend and a hierarchy of communication and analysis agents. Each of the analysis agents, i.e., the nodes of the agent hierarchy, searches autonomously for inefficiencies in a subset of the application processes.

Scalasca

SCALASCA is an automatic performance evaluation system for MPI, OpenMP, SHMEM, and hybrid applications written in C/C++ or Fortran.  SCALASCA generates event traces from running applications and automatically searches them off-line for execution patterns indicating inefficient performance behavior.
 

Tau

TAU  is a tool framework for performance instrumentation, measurement and analysis of parallel programs.  TAU can be configured to capture call-path profiles or phase profiles. This data is visualized with paraprof. Besides to recollect profile based data, TAU is also able to generate event traces  To view this traces TAU has a set of utilities to export the trace to Jumpshot, Vampir, or Paraver.

Documentation guides and videos can be found on the documentation page

VTune

VTune is a performance analyzer from Intel with call graph and sampling profiling. Works with C/C++, Fortran and Java. VTune supports profiling and evaluation of performance counters for single- and multi-threaded programs on all Intel-based hardware platforms. Support for MPI-parallel programs is limited.

Official site
Documentation from Intel
Documentation from LRZ

Vampir

Vampir provides a framework for event analysis and display.

Official site
Flash tutorial
Documentation from LRZ

gprof

gprof (GNU profiler) is a tool for displaying profiling information that helps to determine which parts of the program take the most computation time. It is part of the GNU binutils package.

Programming models and Libraries

COMPSs programming model

COMPSs is a programming model that enables automatic parallelization of sequential codes keeping the application unaware of the underlying platform. It is aimed to coarse grain parallelization, at the level of simulations or similar that exchange data in files.

FFTW

FFTW (Fastest Fourier Transform in the West) is a library for calculating the discrete Fourier transform (DFT) in one or more dimensions, of arbitrary input size, and of both real and complex data (as well as of even/odd data, i.e. the discrete cosine/sine transforms or DCT/DST). FFTW's performance is typically superior to that of other publicly available FFT software, and is even competitive with vendor-tuned codes while still being portable across architectures.

FFTW is used by Gromacs for accelerated calculation of long-range electrostatic interactions.

Official site
Documentation
Benchmarks

MKL

Intel Math Kernel Library (MKL) is a collection of highly-optimized routines for computationally intensive calculations such as BLAS, LAPACK, FFT etc. They are multi-core/multi-processor ready.

Official site
Documentation

MPI

MPI is a library specification for message-passing, proposed as a standard by a broadly based committee of vendors, implementors, and users. There are various implementations of the MPI implementation.

Official site of the MPI forum
Documentation of the specification
API overview on the clusters at LRZ
Usage of MPI at LRZ
Resources for learning MPI programming
Tutorials

Version Control Systems

SVN

SVN has superceded the old CVS as the standard, open source version control system. Founded in 2000 by CollabNet, Inc., the Subversion project and software have seen incredible success over the past decade. Subversion has enjoyed and continues to enjoy widespread adoption in both the open source arena and the corporate world.

The SVN project homepage
Subversion Community Guide
FAQ
The SVN book

git

 
 

Git is a very powerful disctributed version control system that is widely used by major software projects. The following sites provide detailed information about its usage.

http://git-scm.com: Official site of the project
Manual pages from kernel.org
Git tutorial
Everyday Git
Git User Manual

Video material

Video tutorials on git: http://gitcasts.com/

Linus on git at Google talk - a great presentation from the guru

Visualization Tools

Maestro

Maestro is the unified interface for all Schrödinger software. Impressive rendering capabilities, a powerful selection of analysis tools, and an easy-to-use design combine to make Maestro a versatile modeling environment for all researchers.

Molden

Molden is a package for displaying Molecular Density from the Ab Initio packages GAMESS-UK , GAMESS-US and GAUSSIAN and the Semi-Empirical packages Mopac/Ampac, it also supports a number of other programs via the Molden Format.
 

PyMol

PyMOL is an open-source, user-sponsored, molecular visualization system. It can produce high quality 3D images of small molecules and biological macromolecues, such as proteins. 

PyMOL is one of a few open source visualization tools available for use in structural biology. The Py portion of the name refers to the fact that it extends, and is extensible by the Python programming languaje.

VMD

VMD is a molecular visualization program for displaying, animating, and analyzing large biomolecular systems using 3-D graphics and built-in scripting. VMD  is distributed free of charge, and includes source code.


Tutorials and manuals