PADS/ML A Functional Data Description Language Yitzhak Mandelbaum - - PowerPoint PPT Presentation

▶
pads ml
SMART_READER_LITE
LIVE PREVIEW

PADS/ML A Functional Data Description Language Yitzhak Mandelbaum - - PowerPoint PPT Presentation

Oct 20, 2022 394 likes •734 views

PADS/ML A Functional Data Description Language Yitzhak Mandelbaum Princeton University, AT&T With: Kathleen Fisher, David Walker, Mary Fernandez, Artem Gleyzer Meet Stan Stan works for a large networking company, managing their routers.

slide-1
SLIDE 1

PADS/ML

A Functional Data Description Language

Yitzhak Mandelbaum Princeton University, AT&T

With: Kathleen Fisher, David Walker, Mary Fernandez, Artem Gleyzer

slide-2
SLIDE 2

Meet Stan

2

Stan works for a large networking company, managing their routers. Every day, Stan sends and receives commands to and from his routers.

slide-3
SLIDE 3

Meet Stan

2

Stan works for a large networking company, managing their routers. Every day, Stan sends and receives commands to and from his routers.

Router(config)# ip cef distributed Router(config)# tag-switching tsp-tunnels Router(config)# interface e0/1 Router(config-if)# tag-switching tsp-tunnels Router(config-if)# interface e0/2 Router(config-if)# tag-switching tsp-tunnels Router(config-if)# exit Router(config)# interface tunnel 2003 Router(config-if)# tunnel mode tag-switching Router(config-if)# tunnel tsp-hop 1 10.10.0.12 Router(config-if)# tunnel tsp-hop 2 10.50.0.24 lasthop Router(config-if)# exit Router(config)# interface tunnel 2003 Router(config-if)# no tunnel tsp-hop 2 Router(config-if)# tunnel tsp-hop 1 10.10.0.12 last Router(config-if)# exit

slide-4
SLIDE 4

Stan’s New Hope

  • One day, Stan decides to automate!
  • Stan will request entire config from

router and process it externally.

  • But router configuration files can contain

10s of thousands of commands each; stored in a complex and evolving, ad- hoc format.

3

slide-5
SLIDE 5

Current configuration : 1682365 bytes version 12.0 no service pad service tcp-keepalives-in service password-encryption service compress-config service upgrade all ! hostname fromage ! redundancy mode rpr-plus logging buffered 2000000 debugging logging console informational ! username wallace password 5 1240AF334003 username grommit password 5 1A644AC55FF7 ! hw-module slot 1 provision dynamic ! ! ip subnet-zero ip name-server 12.127.17.83 ip flow-cache timeout active 1 ip cef table hardware resource-failure action punt no ip finger ip vrf 8904 description MPLS-PNT| BOOTKEY| LES-MPLS| route-target export 123413:1492 maximum routes 220 40 ! ip vrf 1023 description ANTI-PESTO S.W.A.T. TEAM| export map To_NY_VPN route-target 100:3 maximum routes 150 80 ! ip receive access-list 2660 ! ! policy-map mis_policy_90:0:0:100_output_12288K class rt_class priority police cir percent 90 conform-action transmit exceed-action drop class class-default bandwidth remaining percent 100 queue-limit 500 ms ...

Meet Stan’s Configs

4

slide-6
SLIDE 6

Current configuration : 1682365 bytes version 12.0 no service pad service tcp-keepalives-in service password-encryption service compress-config service upgrade all ! hostname fromage ! redundancy mode rpr-plus logging buffered 2000000 debugging logging console informational ! username wallace password 5 1240AF334003 username grommit password 5 1A644AC55FF7 ! hw-module slot 1 provision dynamic ! ! ip subnet-zero ip name-server 12.127.17.83 ip flow-cache timeout active 1 ip cef table hardware resource-failure action punt no ip finger ip vrf 8904 description MPLS-PNT| BOOTKEY| LES-MPLS| route-target export 123413:1492 maximum routes 220 40 ! ip vrf 1023 description ANTI-PESTO S.W.A.T. TEAM| export map To_NY_VPN route-target 100:3 maximum routes 150 80 ! ip receive access-list 2660 ! ! policy-map mis_policy_90:0:0:100_output_12288K class rt_class priority police cir percent 90 conform-action transmit exceed-action drop class class-default bandwidth remaining percent 100 queue-limit 500 ms ...

username wallace password 5 1240AF334003 username gromit password 5 1A644AC55FF7 !

Meet Stan’s Configs

4

slide-7
SLIDE 7

Current configuration : 1682365 bytes version 12.0 no service pad service tcp-keepalives-in service password-encryption service compress-config service upgrade all ! hostname fromage ! redundancy mode rpr-plus logging buffered 2000000 debugging logging console informational ! username wallace password 5 1240AF334003 username grommit password 5 1A644AC55FF7 ! hw-module slot 1 provision dynamic ! ! ip subnet-zero ip name-server 12.127.17.83 ip flow-cache timeout active 1 ip cef table hardware resource-failure action punt no ip finger ip vrf 8904 description MPLS-PNT| BOOTKEY| LES-MPLS| route-target export 123413:1492 maximum routes 220 40 ! ip vrf 1023 description ANTI-PESTO S.W.A.T. TEAM| export map To_NY_VPN route-target 100:3 maximum routes 150 80 ! ip receive access-list 2660 ! ! policy-map mis_policy_90:0:0:100_output_12288K class rt_class priority police cir percent 90 conform-action transmit exceed-action drop class class-default bandwidth remaining percent 100 queue-limit 500 ms ...

Meet Stan’s Configs

5

slide-8
SLIDE 8

Current configuration : 1682365 bytes version 12.0 no service pad service tcp-keepalives-in service password-encryption service compress-config service upgrade all ! hostname fromage ! redundancy mode rpr-plus logging buffered 2000000 debugging logging console informational ! username wallace password 5 1240AF334003 username grommit password 5 1A644AC55FF7 ! hw-module slot 1 provision dynamic ! ! ip subnet-zero ip name-server 12.127.17.83 ip flow-cache timeout active 1 ip cef table hardware resource-failure action punt no ip finger ip vrf 8904 description MPLS-PNT| BOOTKEY| LES-MPLS| route-target export 123413:1492 maximum routes 220 40 ! ip vrf 1023 description ANTI-PESTO S.W.A.T. TEAM| export map To_NY_VPN route-target 100:3 maximum routes 150 80 ! ip receive access-list 2660 ! ! policy-map mis_policy_90:0:0:100_output_12288K class rt_class priority police cir percent 90 conform-action transmit exceed-action drop class class-default bandwidth remaining percent 100 queue-limit 500 ms ...

Meet Stan’s Configs

ip vrf 1023 description ANTI-PESTO S.W.A.T. TEAM| export map To_NY_VPN route-target 100:3 maximum routes 150 80

5

slide-9
SLIDE 9

Goals Requirements

Audit Communicate Configure Parsing Analysis XML Conversion Transformation Printing

Stan’s World

6

slide-10
SLIDE 10

Goals Requirements

Audit Communicate Configure Parsing Analysis XML Conversion Transformation Printing

Stan’s World

6

slide-11
SLIDE 11

Goals Requirements

Audit Communicate Configure Parsing Analysis XML Conversion Transformation Printing

Stan’s World

6

slide-12
SLIDE 12

Goals Requirements

Audit Communicate Configure Parsing Analysis XML Conversion Transformation Printing

Stan’s World

6

slide-13
SLIDE 13

Goals Requirements

Audit Communicate Configure Parsing Analysis XML Conversion Transformation Printing

Stan’s World

6

slide-14
SLIDE 14

Goals Requirements

Audit Communicate Configure Parsing Analysis XML Conversion Transformation Printing

Stan’s World

6

slide-15
SLIDE 15

Goals Requirements

Audit Communicate Configure Parsing Analysis XML Conversion Transformation Printing

Stan’s World

6

slide-16
SLIDE 16

Enter: PADS/ML

Description Language Compiler Generic Tool Development Framework

7

Data Description Printer Parser Tool Generator Data Generator

Debug Printer XML Translator

Compiler

...

slide-17
SLIDE 17

Based on ML Types

8

We can describe data like we describe data structures.

slide-18
SLIDE 18

Based on ML Types

8

We can describe data like we describe data structures.

Base types describe atomic portions of data

Data PADS/ML

wallace pstring_ws 53.5637 pfloat 100 pint_FW(3)

slide-19
SLIDE 19

Based on ML Types

8

We can describe data like we describe data structures.

Base types describe atomic portions of data

Data PADS/ML

wallace pstring_ws 53.5637 pfloat 100 pint_FW(3)

Familiar type constructors describe complex data: tuples, records, datatypes, ...

slide-20
SLIDE 20

ip vrf 1023 description ANTI-PESTO S.W.A.T. TEAM| export map To_NY_VPN route-target 100:3 max routes 150 80 ptype ip_vrf_command = Description of “description ” * pstring(‘|’) * ‘|’ | Export of “export map ” * pstring(‘\n’) | Route_target of “route-target ” * pint * ‘:’ * pint | Max_routes of “max routes “ * pint * ‘ ‘ * pint ptype ip_vrf = { header : “ip vrf “ * pint * ‘\n’; commands : ip_vrf_command plist(‘\n’) }

slide-21
SLIDE 21

ip vrf 1023 description ANTI-PESTO S.W.A.T. TEAM| export map To_NY_VPN route-target 100:3 max routes 150 80 ptype ip_vrf_command = Description of “description ” * pstring(‘|’) * ‘|’ | Export of “export map ” * pstring(‘\n’) | Route_target of “route-target ” * pint * ‘:’ * pint | Max_routes of “max routes “ * pint * ‘ ‘ * pint ptype ip_vrf = { header : “ip vrf “ * pint * ‘\n’; commands : ip_vrf_command plist(‘\n’) }

slide-22
SLIDE 22

ip vrf 1023 description ANTI-PESTO S.W.A.T. TEAM| export map To_NY_VPN route-target 100:3 max routes 150 80 ptype ip_vrf_command = Description of “description ” * pstring(‘|’) * ‘|’ | Export of “export map ” * pstring(‘\n’) | Route_target of “route-target ” * pint * ‘:’ * pint | Max_routes of “max routes “ * pint * ‘ ‘ * pint ptype ip_vrf = { header : “ip vrf “ * pint * ‘\n’; commands : ip_vrf_command plist(‘\n’) }

slide-23
SLIDE 23

ip vrf 1023 description ANTI-PESTO S.W.A.T. TEAM| export map To_NY_VPN route-target 100:3 max routes 150 80 ptype ip_vrf_command = Description of “description ” * pstring(‘|’) * ‘|’ | Export of “export map ” * pstring(‘\n’) | Route_target of “route-target ” * pint * ‘:’ * pint | Max_routes of “max routes “ * pint * ‘ ‘ * pint ptype ip_vrf = { header : “ip vrf “ * pint * ‘\n’; commands : ip_vrf_command plist(‘\n’) }

slide-24
SLIDE 24

ip vrf 1023 description ANTI-PESTO S.W.A.T. TEAM| export map To_NY_VPN route-target 100:3 max routes 150 80 ptype ip_vrf_command = Description of “description ” * pstring(‘|’) * ‘|’ | Export of “export map ” * pstring(‘\n’) | Route_target of “route-target ” * pint * ‘:’ * pint | Max_routes of “max routes “ * pint * ‘ ‘ * pint ptype ip_vrf = { header : “ip vrf “ * pint * ‘\n’; commands : ip_vrf_command plist(‘\n’) }

slide-25
SLIDE 25

Language Features

Records for data dependency Constraints for additional properties Datatypes for variation Matches for tagged data Recursion for nested structure Polymorphism for code reuse Lists, speculation, early commit, and more ...

10

slide-26
SLIDE 26

11

Types as Modules

Compiler converts PADS/ML types into O’Caml modules. PADS/ML O’Caml Monomorphic type Module Recursive type Recursive Module Polymorphic type Functor (module → ...)

slide-27
SLIDE 27

Generic Tool Framework

Developer writes tool in type-directed manner. Compiler generates a single-pass traversal. User easily applies any tool to any data source.

12

slide-28
SLIDE 28

Generic Tool Framework

Developer writes tool in type-directed manner. Compiler generates a single-pass traversal. User easily applies any tool to any data source.

12

Stan’s Configs Data Format Data Format

...

slide-29
SLIDE 29

Generic Tool Framework

Developer writes tool in type-directed manner. Compiler generates a single-pass traversal. User easily applies any tool to any data source.

12

XML Converter Debug Printer Generic Tool

...

Stan’s Configs Data Format Data Format

...

slide-30
SLIDE 30

Generic Tool Framework

Developer writes tool in type-directed manner. Compiler generates a single-pass traversal. User easily applies any tool to any data source.

12

XML Converter Debug Printer Generic Tool

...

Stan’s Configs Data Format Data Format

...

Traversal Traversal Traversal

...

slide-31
SLIDE 31

Generic Tool Framework

Developer writes tool in type-directed manner. Compiler generates a single-pass traversal. User easily applies any tool to any data source.

12

XML Converter Debug Printer Generic Tool

...

Stan’s Configs Data Format Data Format

...

Traversal Traversal Traversal

...

slide-32
SLIDE 32

Summary

  • A functional description

language for data analysts.

  • A practical framework for

rapidly developing new data processing tools.

  • Based on solid theoretical

foundation.

  • DDC + polymorphic types

[POPL ’06, Thesis].

Describes real formats

  • Stan’s config files
  • Newick tree-structured data
  • AT&T billing data
  • Web server logs
  • ...

Download www.padsproj.org

slide-33
SLIDE 33

The End