in cooperation with

Communication Networks

IBPM: An Open-Source-Based Framework for InfiniBand Performance Monitoring Michael Hoefling1, Michael Menth1, Christian Kniep2, Marcus Camen2

Background: InfiniBand (IB)     

State-of-the art communication technology for interconnection in high-performance computing data centers Point-to-point bidirectional links High throughput (40 Gbit/s with QDR) Low latency Dynamic on-line network reconfiguration

Rate Measurement in IB Networks

IBPM: Demo Overview ibsim-Based Network Simulation  ibsim simulates an IB network  Simple topology changes possible (GUI)  ibsim limitations  No performance simulation possible  No data rate changes possible Real IB Network  Physical network  Allows performance measurements  GUI controlled traffic scenarios

Idea  Extract raw network information from IB network  Analyze output  Derive statistics about performance of the network Topology Extraction  Subnet discovery using ibnetdiscover  Produces human readable file of network topology  Process output to produce graphical representation of the network Remote Counter Readout  Each port has its own set of performance counters  Counters measure, e.g., transferred data, congestion, errors, link states changes

IBPM: Open-Source-Based InfiniBand Performance Monitoring Features  Automatic topology extraction and visualization  Visualization of traffic locality  Visualization of link utilization  Visualization of congestion  Visualization of port performance history Architecture

IBPM: Demo Scenarios Scenario 1: Topology Changes  Node and/or switch becomes unavailable  Connectivity state is represented in the topology map Scenario 2: Port Performance and Link Utilization  Nodes communicate with each other  Port performance accessible through simple point-and-click interface on a node or switch  Link utilization is visualized through utilization-based-coloring of the links in the performance map Scenario 3: Traffic Locality  Nodes use pre-defined traffic patterns  Traffic locality is visualized through locality-based-coloring of the switches in the locality map Michael Hoefling, Michael Menth, Christian Kniep, and Marcus Camen: "IBPM: An Open-Source-Based Framework for InfiniBand Performance Monitoring", in Proceedings of the 16th GI/ITG Conference on Measurement, Modeling, and Evaluation of Computer and Communication Systems (MMB) and Dependability and Fault Tolerance (DFT), March 2012, Kaiserslautern, Germany

These authors are with the University of Tuebingen, Tuebingen, Germany. 2 These authors are with science+computing ag, Tuebingen, Germany. 1

University of Tuebingen · Sand 13 · 72076 Tübingen Phone: +49-7071-29-70507 [email protected] http://kn.inf.uni-tuebingen.de/staff/hoefling