Open  Resilient  Cluster  Manager   (ORCM)  

Ralph  H.  Castain,  Ph.D.  

1

Objec
–  Easily  customized,  extended   –  Replace/override  any  behavior   –  Support  proprietary  as  well  as  open  extensions   –  Fully  u
•  Establish  ecosystem  

–  Academic,  industry  collaborators,  OMPI-­‐like   community  

•  Provide  a  reference  solu
–  Publicly  available,  performant,  flexible,  scalable   –  Easily  replace  any  pieces   2  

2003  -­‐  present   OMPI/ORTE  

10s  of  Knodes   Open  MPI  

OpenRTE  

Enterprise   Router  

Cisco  

Cluster   Monitor  

EMC  

SCON/RM  

Intel  

ORCM/SCON   3

ORCM  Roadmap   •  Monitoring  system  (1Q2015)   –  System  environment,  power,  process  usage,  etc.  

•  Overlay  network/pub-­‐sub  (3Q2015)   –  More  in  a  minute  

•  Job  launch  (3Q2015)   –  Can  do  it  now,  but  need  support  for  all  MPIs  

•  Workload  manager  (2016)   –  Lightweight   4  

Hierarchical  Arch   SMC  

Row  

Row  

Rack  

Rack  

Rack  

Rack  

Rack  

Rack  

CN   CN  

CN   CN  

CN   CN  

CN   CN  

CN   CN  

CN   CN  

CN   CN  

CN   CN  

CN   CN  

CN   CN  

CN   CN  

CN   CN   5  

Integra
•  Network   –  QoS  controls   –  Sta
•  Power   –  Various  modes,  dynamic  controls,  site-­‐level  control   6

Instant  On  Steps   •  Prestage  executables  to  IO  nodes   •  Allocate  and  launch   •  • 

Launch  message  =>  orte_job_t,  included  in  alloca
•  Distributed  mapping/rolling  start  (branch)  

–  Each  daemon  computes  map,  stores  all  data  (map,  endpoints,  network   topo)  in  shared  memory  region  for  job   –  Connect/accept  =>  pass  SM  connec
•  Eliminate  modex  

–  Sta
•  Eliminate  fence  at  end  of  mpi_init  

–  Modex-­‐recv  becomes  flag  that  proc  is  ready   –  RM  flags  all  procs  on  node  upon  first  request  so  subsequent  checks   are  local   7

Defini
•  In-­‐flight  analy
Requirements   •  Scalable  to  exascale  levels   –  Beier-­‐than-­‐linear  scaling  of  broadcast  

•  Resilient   –  Self-­‐heal  around  failures   –  Reintegrate  recovered  resources  

•  Dynamically  configurable   –  Sense  and  adapt,  user-­‐directable   –  On-­‐the-­‐fly  updates  

•  Open  source  (non-­‐GPL)   9

High-­‐Level  Architecture   SCON-­‐MSG  

RMQ  

ZMQ  

Send/Recv  

BTL  

STL   Portals4   IB   UGNI  

SM   USNIC  

OOB   TCP   CUDA  

UDP   TCP  

SCIF   SCON-­‐ANALYTICS  

FILTER  

AVG  

Workflow/Pub-­‐Sub  

THRESHOLD  

10  

Messaging  APIs   •  Typical  send/recv   –  Non-­‐blocking,  iovec  or  buffered  (built-­‐in  heterogeneous  support)  

•  Open  channel   –  Specify  remote  peer  and  endpoint  tag   –  Provide  hints  on  type  of  data  messaging  to  be  used   •  Stream,  command/control,  etc.  

–  Specify  desired  quality  of  service   •  Guaranteed  delivery  of  every  message,  high  priority,  etc  

•  Subscribe  to  data  stream   –  Specify  source  and  data  

11  

Message  Rou
•  Defined  per  transport  (branch)   •  Heals  routes   –  Provides  alternate  route  upon  failure   –  Up-­‐level  error  if  no  alternate  available  on  this   transport   –  Allows  re-­‐rou
Message  Reliability   •  Plugin  architecture   –  Selected  per  transport,  requested  quality  of  service  

•  ACK-­‐based  (cmd/ctrl)   –  Ack  each  message,  or  window  of  messages,  based  on  QoS   –  Resend  or  return  error  –  QoS  specified  policy  and  number  of  retries  before   giving  up  

•  NACK-­‐based  (streaming)   –  Nack  if  message  sequence  number  is  out  of  order  indica
•  Mul
Analy
•  Event  genera

Open Resilient Cluster Manager (ORCM) - GitHub

Hierarchical Arch. 5. SMC. Row. Row. Rack. CN CN. CN CN. Rack. CN CN. CN CN ... Each daemon computes map, stores all data (map, endpoints, network.

413KB Sizes 9 Downloads 85 Views

Recommend Documents

Actionable = Cluster + Contrast? - GitHub
pared to another process, which we call peeking. In peeking, analysts ..... In Proceedings of the 6th International Conference on Predictive Models in Software.

trouble ticket manager - GitHub
Department of Computer Science and Information Technology ... Different software tools have been developed and are in use in order to handle and ..... Since this application filters trouble tickets and automate based on the contents .... or e-busines

Open Adventure - GitHub
Sonic attack is considered 20 times louder than speaking volume. Any characters (except you) within range must succeed at a will save or suffer 1 stun counter.

Open Adventure - GitHub
Page 1. Open Adventure. Adventure Pack. Page 2. Equipment Cards. Page 3. Equipment Cards. Page 4. Page 5.

Open Modeling Framework - GitHub
Prepared for the U.S. Department of Energy, Office of Electricity Delivery and Energy Reliability, under Contract ... (ORNL), and the National Renewable Energy.

Open Data Canvas - GitHub
Top need for accessing data online. What data is most needed? Solution. How would you solve this problem? ... How big is the universe of users? Format/Use.

Open Data publishing method framework - GitHub
Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/. Build. Assess. Iterate. Certification. Publish!

Open Source Code Serving Endangered Languages - GitHub
ten called low-resource, under-resourced, or minority lan- guages) ... Our list is updatable more ... favorites on other social media sites, and, generally, a good.

Slow Dynamics and Thermodynamics of Open ... - APS Link Manager
Aug 2, 2017 - which, differently from quasistatic transformations, the state of the system is not able to continuously relax to the equilibrium ensemble.