National Energy Research Scientific Computing Center (NERSC) - - PowerPoint PPT Presentation

▶
national energy research scientific computing center nersc
SMART_READER_LITE
LIVE PREVIEW

National Energy Research Scientific Computing Center (NERSC) - - PowerPoint PPT Presentation

Dec 18, 2023 723 likes •882 views

National Energy Research Scientific Computing Center (NERSC) Highly Scalable Networking Configuration Management For Highly Scalable Systems Nicholas P. Cardo NERSC Center Division, LBNL CUG 2008, Helsinki N ATIONAL E NERGY R ESEARCH S

slide-1
SLIDE 1

National Energy Research Scientific Computing Center (NERSC)

Highly Scalable Networking Configuration Management For Highly Scalable Systems

Nicholas P. Cardo NERSC Center Division, LBNL CUG 2008, Helsinki

slide-2
SLIDE 2

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

2

Setting the Stage

  • Systems are getting large
  • Systems are getting more complex
  • Networking capability is increasing
  • Systems have many nodes
  • Systems have many network interfaces
  • Systems have many different interface types
slide-3
SLIDE 3

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

3

Cardo’s Corollary #3 If it can change, it will change.

slide-4
SLIDE 4

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

4

Network Files

  • /etc/sysconfig/network/ifcfg-eth0
  • /etc/sysconfig/network/ifcfg-eth1
  • /etc/sysconfig/network/ifcfg-ss
  • /etc/sysconfig/network/scripts/
slide-5
SLIDE 5

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

5

Standard Interface Configuration

  • cd /etc/sysconfig/network
  • vi ifcfg-eth0

– Set IPADDR – Set NETMASK – Set MTU – Set other parameters

For each interface. On each node.

slide-6
SLIDE 6

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

6

Now on the Cray

  • ssh root@boot
  • xtopview -n <nid>
  • cd /etc/sysconfig/network
  • xtspec ifcfg-eth0
  • vi ifcfg-eth0

– Set IPADDR – Set NETMASK – Set MTU – Set other parameters

  • exit
  • c
  • <some text>
  • .

Repeat per interface Repeat per node

slide-7
SLIDE 7

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

7

Lets Do the Math

56 Nodes 92 Interfaces 5 minutes per change 1 SS per node 20 10Gb 16 1Gb @ 5 minutes = 460 minutes = 7 hours 40 minutes

slide-8
SLIDE 8

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

8

Goals

  • Single points of data entry for routing and interface

configurations.

  • Scalable to support all service nodes.
  • Flexible to handle complex routing if necessary.
  • Minimal specialization.
slide-9
SLIDE 9

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

9

The Setup

  • ssh root@boot
  • xtopview
  • mkdir /etc/NERSC
  • cd /etc/sysconfig
  • mv ifcfg-ss /etc/NERSC
  • mv ifcfg-eth0 /etc/NERSC
  • ln -s /etc/NERSC/ifcfg-eth0 ifcfg-eth0
  • ln -s /etc/NERSC/ifcfg-ss ifcfg-ss
slide-10
SLIDE 10

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

10

Interface Configuration File

# Node Location Type Dev Address Netmask MTU nid04100 c1-0c0s1n0 login01 eth0 128.55.81.34 255.255.248.0 9000 nid04100 c1-0c0s1n0 login01 ss 128.55.42.134 255.255.255.192 NA

IPADDR NETMASK MTU Needed in ifcfg-<int>

slide-11
SLIDE 11

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

11

# Node Location Type Dev Route Gateway Netmask MTU Fwd nid04100 c1-0c0s1n0 login01 eth0 default 128.55.80.1 NA NA 0 nid04100 c1-0c0s1n0 login01 ss 128.55.32.0 128.55.42.130 255.255.224.0 9000 0

/sbin/route add $TYP $ADR gw $GWY $MSK dev $DEV $MTU

  • net or -host

echo $FOR >/proc/sys/net/ipv4/ip_forward

Route Configuration File

Do this in POST_UP_SCRIPT

slide-12
SLIDE 12

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

12

ARP Table Pre-Load

/sbin/arp -f /etc/NERSC/arp-ss -i ss 128.55.42.130 00:00:10:00

Do This in POST_UP_SCRIPT

slide-13
SLIDE 13

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

13

Summary

  • One file text describes all interfaces
  • One file text file describes all routes
  • One time setup
slide-14
SLIDE 14

NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER

14

Questions?