HPC-SIG Ecosystem Validation Jan. 14 2019 Baptiste Gerondeau - - PowerPoint PPT Presentation

HPC-SIG Ecosystem Validation

• Jan. 14 2019

Baptiste Gerondeau Renato Golin

For more info visit

linaro.org/hpc

HPC-SIG Lab and Validation Matrix

Aggregate machines in the same infrastructure, and validate their performance using a Validation Matrix

• Validation Matrix must be applicable to every machine
• Validation Matrix dimensions are software configurations

To generate as few tests as possible, we need to simplify the matrix without losing information

For more info visit

linaro.org/hpc

HPC-SIG Lab’s Infrastructure

The infrastructure needs to :

• Dispatch jobs (tests, provisioning, benchmarks)
• Provide DHCP/TFTP services
• Provide Package Cache services
• Provide a secure file/results storage service
• Be Low Maintenance
• Be able to be replicated anywhere else

For more info visit

linaro.org/hpc

Simplifying Infrastructure

Identifying the different dimensions

A Vertical Slice of the Stack Principal dimensions : ➔ Application ➔ HPC environment stack ➔ Machine provisioning

• HPC Stack : OpenHPC
• Validation Application :

OpenHPC’s testsuite

For more info visit

linaro.org/hpc

Simplifying Infrastructure

Identifying the different dimensions

The Stack from the Lab’s point of view Machine provisioning : ➔ Network configuration ➔ Kernel ➔ OS ➔ HPC Stack

• Multiple ways to do the

provisioning

For more info visit

linaro.org/hpc

Simplifying Infrastructure

Identifying the different dimensions

Provisioning Method Variations Multiple ways to provision : ➔ Warewulf Stateless (VNFS) ➔ Warewulf Stateful (OS image) ➔ Ansible

For more info visit

linaro.org/hpc

Simplifying Infrastructure

Identifying the different dimensions

Different Network Layouts

• Flat : Machines reachable from anywhere
• Tree: Machines reachable from cluster

head node only

• Root : Master with DHCP/TFTP server

For more info visit

linaro.org/hpc

Simplifying Infrastructure

Identifying the different dimensions

Different Kernels

• Upstream from OS
• ERP : Entreprise Reference

Platform

• Contains support for

platforms in the process of being upstreamed

For more info visit

linaro.org/hpc

Simplifying Infrastructure

Identifying the different dimensions

Different Operating Systems

• 3 OSes available to the user
• No Debian support in OpenHPC

For more info visit

linaro.org/hpc

Simplifying Infrastructure

Abstractions, and the user’s environment

Abstracting Network Variations

• Invisible to the user
• Handled by the lab installer
• Dependent on hardware

For more info visit

linaro.org/hpc

Simplifying Infrastructure

Abstractions, and the user’s environment

Abstracting Provisioning Variations

• Multi-staged provisioning
• Coexistence
• Dependent on hardware

For more info visit

linaro.org/hpc

Simplifying Infrastructure

Abstractions, and the user’s environment

Abstracting Environment Variations

• Control over HPC Stack
• Common OS configuration
• Idempotency
• Package Caches

For more info visit

linaro.org/hpc

Simplifying Infrastructure

Abstractions, and the user’s environment

Accounting for extra HPC services

• Infiniband Support
• Lustre server support
• Future additional features

(additional hardware)

For more info visit

linaro.org/hpc

Simplifying Infrastructure

What the User sees, configures

The Lab’s Interface ➔ Choose Application ❖ Lab picks default configuration ❖ User fine tunes configuration

For more info visit

linaro.org/hpc

Validation matrix

Cluster Deployment

For more info visit

linaro.org/hpc

Validation matrix

Distributed Applications Enablement

For more info visit

linaro.org/hpc

Validation matrix

Toolchain Benchmarking

For more info visit

linaro.org/hpc

Validation matrix

Library Enablement and Enhancement

For more info visit

linaro.org/hpc

Future

• Vendors to rely on Linaro for base OSS validation

○ We have multiple vendors available ○ On a standardised infrastructure

For more info visit

linaro.org/hpc

Future

• Vendors to rely on Linaro for base OSS validation

○ We have multiple vendors available ○ On a standardised infrastructure

• Share our work

○ OpenHPC Ansible recipes (with the OpenHPC community) ○ SDI (MrP, Jenkins, Ansible) helping members to replicate our work ○ Community CI (OpenHPC test-suite, MPI MTT, OpenMP tests, OpenBLAS CI)

For more info visit

linaro.org/hpc

Future

• Vendors to rely on Linaro for base OSS validation

○ We have multiple vendors available ○ On a standardised infrastructure

• Share our work

○ OpenHPC Ansible recipes (with the OpenHPC community) ○ SDI (MrP, Jenkins, Ansible) helping members to replicate our work ○ Community CI (OpenHPC test-suite, MPI MTT, OpenMP tests, OpenBLAS CI)

• Allow our engineers to develop the ecosystem

○ Internal tests and benchmarks (via Jenkins, no infrastructure knowledge needed) ○ Testing new packages, libraries, compilers (comparison jobs, CI results, statistic analysis)

For more info visit

linaro.org/hpc

Future

• Vendors to rely on Linaro for base OSS validation

○ We have multiple vendors available ○ On a standardised infrastructure

• Share our work

○ OpenHPC Ansible recipes (with the OpenHPC community) ○ SDI (MrP, Jenkins, Ansible) helping members to replicate our work ○ Community CI (OpenHPC test-suite, MPI MTT, OpenMP tests, OpenBLAS CI)

• Allow our engineers to develop the ecosystem

○ Internal tests and benchmarks (via Jenkins, no infrastructure knowledge needed) ○ Testing new packages, libraries, compilers (comparison jobs, CI results, statistic analysis)

HPC Lab Setup https://github.com/Linaro/hpc_lab_setup Ansible OpenHPC installation recipe : https://github.com/Linaro/ansible-playbook-for-ohpc

Thanks!