CWB Network Information exChange Environment (NICE) Mark Cheng - - PowerPoint PPT Presentation

2011/4/1 CWB NICE - Mark Cheng 1

CWB

Network Information exChange Environment

(NICE)

Mark Cheng Central Weather Bureau, Taiwan, R.O.C.

2011/4/1 CWB NICE - Mark Cheng 2

NICE A network based application framework for real time distributed data processing

1. Roles and Responsibilities of CWB 2. Meteorological data processing requirements 3. HW environment and application systems 4. NICE requirement and components 5. Operation concepts 6. Real world applications and future

2011/4/1 CWB NICE - Mark Cheng 3

• Location

land-sea boundary(121E,24N)

• Size

400km × 150km

• Topography

3/4 mountain area central mountain ridge : 2000m avg strong terrain forcing

• Sever weather phenomena

spring : MeiYu summer : Typhoon, sounder storm winter : cold surge

• Forecast Challenges

Typhoon & Heavy rain fall

Key issues to TW weather forecast challenge

Complex terrain, intersection of continent Asia and Pacific ocean, intersection of tropical and sub-tropical monsoon area, prone to have typhoon, heavy rain fall, cold surge and drought

MEGI

2011/4/1 CWB NICE - Mark Cheng 4

衛星 / 雷達 / 降雨 / 能見 溫度 / 氣壓 / 溼度 / 風力 觀測、警報、特報、預報 守視、天氣、氣候 颱風、豪雨、寒潮、濃霧 地震、海嘯、波浪、潮汐

CWB

Met.、Mar.、seis. Hazard Mitigation

颱風/暴雨/乾旱 洪水/土石/搜救

Environmental Conservation

環保/森林/水保 陸域/海域

Resource Planning

水利/能源/農業 漁牧/海域

Governmental

Applications

國防/民航/海巡 環保/水庫/保建

• Int. Tech.

Cooperations

國內外學研單位 國際作業中心

Living Service

生活/旅遊/交通 颱風/暴雨/寒潮

Value Added Applications

保險/物流/產銷 營建/媒體/交通

• Int. Data

Exchange

國防/民航/環保 國際作業中心

urgent

sustainablility Econo. General Suppo. Devlopment Operation

Government Service Public Service I nternational Collaborations

Econo.

Roles and Responsibilities of CWB

2011/4/1 CWB NICE - Mark Cheng 5

Courtesy：ECMWF

Global Data Acqusition Model Forecast Information Display Verification Product Generation Forecast Decision Data processing

2011/4/1 CWB NICE - Mark Cheng 6

NWP HPCs MSS WINS Satellite, Rader

Router

Rain Gauge Mesonet

Giga Switch Ethernet

MIPS

WAN LAN

Internet FireWall FireWall

• Admin. staff

Government Agencies

security

FireWall

AMDP Grapher

HW & Network for CWB Weather Information Processing

2011/4/1 CWB NICE - Mark Cheng 7

Graphic output 資 acqusition acqusition decode decode error check analysis

Atomspheric dynamic theory High performance computer Atmospheric structure & phenomena forecast future change 原理 運算 分析 推算

Other centers Research agency

料

Numerical Weather Prediction System Concepts

2011/4/1 CWB NICE - Mark Cheng 8

CWB NWP Models in

• peration

Model Resolution Run period( length) # Mbr Model days GFS T239-L30 00z(120h),12z(216h) 1 14 GFS T319-L40 00z(120h),12z(216h) 1 14 NFS 45/15/5km-L30 00z/06z/12z/18z(84h) 2 28 WRF 45/15/5km-L45 00z/06z/12z/18z(84h) 2 28 TWRF 45/15/5km-L45 00z/06z/12z/18z(84h) 1 14 WRF 20/4km-L45 00z/06z/12z/18z(84h) 1 14 LAPS 9km-L31 every 3 hours(12h) 4 16 GCFS T42-L18 1st(210d), 16th(210d) 40 560 RCFS 60km-L18 10th(150d) 30 150 total 838

2011/4/1 CWB NICE - Mark Cheng 9 Grid data processing Grid product service Weather service Guidance product service MesoNet OBS Rain gauge OBS Satellite OBS Radar network Lighting OBS

WINS display

Professional product service

Numerical Weather Prediction

Forecaster Seismic OBS Marin forecast Local weather station Media service Public service Grapher Graphic product service University CAF, Navy, Army EPA CAA CAF, Navy

GTS

Internet WWW

Internet Data

Forecaster

GTS data collection Weather code service AMDP OBS data service University Research

Navy EPA CAA

Government Organizations Emergency Management Statistical forecast

Flow for Major Data Processing Systems

University CAF, Navy EPA

2011/4/1 CWB NICE - Mark Cheng 10

The need for an integrated software development environment and application framework to

• Support the development and operation of various real time

distributed meteorological data processing systems.

• Reduce

the efforts in developing and maintaining similar automation mechanism for different application systems. (maximize reusability)

• Integrate past experiences and best practices in system operation

mechanism development become an unified architecture and tool kits for optimal system operations.

• Utilizing open system software technologies, components &

standards to speed up AP development while maintaining the system’s operability, flexibility, reliability and interoperability.

2011/4/1 CWB NICE - Mark Cheng 11

Network Information exChange Environment

NICE

Information Transport Information Service User Interface Processes Automation Monitoring Logging MSS Interface Data Access Management High Avail. Fail over HPC Interface Applications Data / Info. NICE Applications Application Operators

MSS HPC

• nline

storage non NICE Applications Application Configurations

• ffline

storage

2011/4/1 CWB NICE - Mark Cheng 12

NICE components (1/3)

(1)Information Transport facility - IT

Provides the solutions for reliably exchange of data and/or messages in different hosts within or among systems across network. Access controls can be applied to information objects and hosts if so wished.

(2)Process Automation facility - PA

Provides the solutions for automated processes management. The process activation mechanisms include : time driven (one time or periodical), event driven (single event or logical combination of multiple events). The management functions include : creation / deletion / rerun, process queueing, # of processes, execution elapse time, …

(3)Data Access and Management facility - DA/DM

Provides the solutions for on-line data records and files access in local or remote hosts across network. Files can be either plane or

• indexed. File management mechanism includes : creation, purging,

archiving, backup, restore.

2011/4/1 CWB NICE - Mark Cheng 13

NICE components (2/3)

(4)Information Service facility - IS

Provides the solutions for reliably receiving and sending information objects with standard FTP, RCP & LDM protocols to interface the outside world. Status detection, Timeout, retry, multithreading and queueing mechanisms are provided for reliable transport management.

(5)System MONitoring & LOGging facility – MON/LOG

Provides the solutions for detecting, monitoring, logging & reporting resource utilization and/or processes activities events of application systems. Events include : resource usage & utilization, data availability, network connectivity, process life cycle, activities timing. All events can be centralized managed and displayed in an operator UI.

(6)System Fail Over facility - FO

Provides the solutions for maintaining high availability of system

• peration. Critical resources and services can be configured, monitored,

managed and failovered by a decision making kernel with the support of activity monitoring, abnormal detection and fault isolation.

2011/4/1 CWB NICE - Mark Cheng 14

NICE components (3/3)

(7)User Interfacing facility - UI

Provides the solutions for users and operators to interface with the

• system. A set of X-window based widgets include : menu items, selection

list, command button, file selection, text entry, dialog box & message window can be configured and composed via a scripting language for system’s specific needs.

(8)MSS interfacing facility - MSS

Provides the solutions for interfacing with the Hierarchical Storage Management system in CWB. Storage pool inventory query, migration/de- migration control and basic file operations can be applied to file objects in local and/or remote storage devices and/or servers.

(9)HPC interfacing facility - HPC

Provides the client side solution for interfacing with the High Performance Computer system in CWB. A Simplified commands set to encapsulate different batch systems commands for most commonly used commands for Job submission, execution control & status query.

2011/4/1 CWB NICE - Mark Cheng 15 Service processor Service processor Service processor Service processor Service processor Service processor Service processor Service processor Service processor Service processor Service processor N I C E P R O T O C O L S N E T W O R K

Configuration／Logging／Monitoring／Operation Interface

: NICE process : AP process : Data flow : Control flow

~ ~

Distributed Data store Distributed

• Prod. store

Temporary Data store

~ ~

O T H E R P R O T O C O L S

channel processor message processor data processor channel connect Activities monitor message router information Service File manager process manager Data store Message store

High Availability Fail Over control

protocol converter Prod. store Service processor

NICE Application Framework & Operation Concept

2011/4/1 CWB NICE - Mark Cheng 16

Grid Architecture

defined by Ian Foster etc.

MIPS/AMDP/NWP/GRAPH/RELAY/WINS/… Application

PA/DM(FP/BK/AR)/IS/FO/UI Collective IT(Nlap)/DA/MON/MSS/HPC/ Resource IT(IR/Nr/Ns/DT/Ndq/DIO/HS) Connectivity Computer/Storage/Network resources Fabric NI CE GRI D

Application Transport Internet Link

I nternet

Coordinating multiple resources

• btain the information &negotiate access

Communicating easily and securely user applications that utilizing under layer services Shared accessible resources

2011/4/1 CWB NICE - Mark Cheng 17

NICE Application development concepts

Application Interface : C/Fort api , commands/utilities , high level GUI

Config. Config. Config.

Nlap GetPrimary Information Transport NpaEm DM-DA DM-FM

Information Service NmonDtcResident

LDM NmqSvr NlapSvr

Config.

Nmon/Nlog

Config.

CC DT MT GW

Active Backup mode HA Fail Over control Config.

DIO

pq NpqFwd upq Pqact Config. Config. file Ap Process Ap Process Ap Process

2011/4/1 CWB NICE - Mark Cheng 18

NICE LAP operation concepts

Service processor

AP-A Primary Host

NLAP SERVER

Service processor

AP-A Backup Host Logical AP Service Info.

Service processor

AP-B Primary Host

Service processor

AP-B Backup Host

Service processor

AP-C Service Host A Active AP Host Info.

Service processor

AP-C Service Host B

Service processor

AP-C Service Host C

(1) Regist AP-C Service location

(3) Use active host for data communications (3) Talk to available services (3) Talk to specific host

2011/4/1 CWB NICE - Mark Cheng 19

NICE Information Service operation concepts

Message Router Resend control Send control Receive control Config File Poller Config File Resend queue Ldm, FTP, RCP, User protocol Send queue Up stream systems Received Data File

Service processor

receive processor Down stream systems Data File To be Transmited Sender Sender Sender Send queue Config File

Service processor

send processor

2011/4/1 CWB NICE - Mark Cheng 20

S-1 Stand Alone (Single) S-0 Stand By (Backup) S-2 Primary (Active) Initial (b) Peer Faults (Start AP) (f) Forced to go backup (e) Peer Ok (Go primary) (i) Peer Faults (c) Local Faults

• r Network Down
• r peer is A or S

(g) Peer Faults or Network Down (j) Peer OK or Network Down Events (Actions) (a) (h) Local Faults (Stop AP) (d) Local Faults(Stop AP)

NICE HA/FO operation concepts

Active-Backup mode

2011/4/1 CWB NICE - Mark Cheng 21 Service processor Service processor

Status Collector Status Detector Monitor UI Status manager Status Logger

Warning message

NICE Mon/Log operation concepts

AP System

• perator

Detecting Config. Devices & Resources

Service processor

AP processes Devices & Resources Devices & Resources Log files Managing policy

Service processor Service processor Service processor

AP processes

2011/4/1 CWB NICE - Mark Cheng 22

Real world applications and future

The NICE has been exist, utilized and continuously improved for about 10 years in CWB. Currently, 147 hosts of 39 systems were installed and running some components of NICE. Every day, over 1.2 million information objects are routed by NICE message routers and over 150 thousand processes are managed by NICE process managers. The storage volume managed by NICE file manager around 20TB and the data access transaction per day over 6 million records(files) while the transaction volume well exceed 10TB. The product files delivered by information service to end users and/or down stream systems outside CWB are

• ver 35 thousand files and exceed 100GB.

Still, the NICE needs to be enhanced to increase it’s support to web browser based UI and Linux HA based fail over control. The NICE also needs to implement the support for the new HPC to be procured in the next few years.

2011/4/1 CWB NICE - Mark Cheng 23

Thanks for your attention Questions？