in cooperation with

Communication Networks

IBPM: An Open-Source-Based Framework for InfiniBand Performance Monitoring Michael Hoefling1, Michael Menth1, Christian Kniep2, Marcus Camen2

Background: InfiniBand (IB)     

State-of-the art communication technology for interconnection in high-performance computing data centers Point-to-point bidirectional links High throughput (40 Gbit/s with QDR) Low latency Dynamic on-line network reconfiguration

Rate Measurement in IB Networks

IBPM: Demo Overview ibsim-Based Network Simulation  ibsim simulates an IB network  Simple topology changes possible (GUI)  ibsim limitations  No performance simulation possible  No data rate changes possible Real IB Network  Physical network  Allows performance measurements  GUI controlled traffic scenarios

Idea  Extract raw network information from IB network  Analyze output  Derive statistics about performance of the network Topology Extraction  Subnet discovery using ibnetdiscover  Produces human readable file of network topology  Process output to produce graphical representation of the network Remote Counter Readout  Each port has its own set of performance counters  Counters measure, e.g., transferred data, congestion, errors, link states changes

IBPM: Open-Source-Based InfiniBand Performance Monitoring Features  Automatic topology extraction and visualization  Visualization of traffic locality  Visualization of link utilization  Visualization of congestion  Visualization of port performance history Architecture

IBPM: Demo Scenarios Scenario 1: Topology Changes  Node and/or switch becomes unavailable  Connectivity state is represented in the topology map Scenario 2: Port Performance and Link Utilization  Nodes communicate with each other  Port performance accessible through simple point-and-click interface on a node or switch  Link utilization is visualized through utilization-based-coloring of the links in the performance map Scenario 3: Traffic Locality  Nodes use pre-defined traffic patterns  Traffic locality is visualized through locality-based-coloring of the switches in the locality map Michael Hoefling, Michael Menth, Christian Kniep, and Marcus Camen: "IBPM: An Open-Source-Based Framework for InfiniBand Performance Monitoring", in Proceedings of the 16th GI/ITG Conference on Measurement, Modeling, and Evaluation of Computer and Communication Systems (MMB) and Dependability and Fault Tolerance (DFT), March 2012, Kaiserslautern, Germany

1 2

These authors are with the University of Tuebingen, Tuebingen, Germany. These authors are with science+computing ag, Tuebingen, Germany.

University of Tuebingen · Sand 13 · 72076 Tübingen Phone: +49-7071-29-70507 [email protected] http://kn.inf.uni-tuebingen.de/staff/hoefling