Building Business Systems with DSLs atop OpenResty - - PowerPoint PPT Presentation

▶
building business systems with dsls atop openresty
SMART_READER_LITE
LIVE PREVIEW

Building Business Systems with DSLs atop OpenResty - - PowerPoint PPT Presentation

Dec 10, 2022 204 likes •1.07k views

Building Business Systems with DSLs atop OpenResty agentzh@openresty.org Yichun Zhang (@agentzh) 2016.9 NGINX + LuaJIT The all-inclusive philosophy Simple Small Fast Flexible Synchronously nonblocking Light threads &

slide-1
SLIDE 1

Building Business Systems with DSLs atop OpenResty ☺agentzh@openresty.org☺

Yichun Zhang (@agentzh)

2016.9
slide-2
SLIDE 2
slide-3
SLIDE 3
slide-4
SLIDE 4
slide-5
SLIDE 5

♡ NGINX + LuaJIT

slide-6
SLIDE 6

♡ The all-inclusive philosophy

slide-7
SLIDE 7

Simple Small Fast Flexible

slide-8
SLIDE 8

Synchronously nonblocking

slide-9
SLIDE 9

Light threads & semaphores

slide-10
SLIDE 10

Cosockets

slide-11
SLIDE 11

Timers and Sleeping

slide-12
SLIDE 12

Shm-based dictionaries and queues

slide-13
SLIDE 13

Dynamic SSL Handshakes

slide-14
SLIDE 14
slide-15
SLIDE 15

Dynamic Load Balancers

slide-16
SLIDE 16

upstream { server 0.0.0.1; # just a place holder balancer_by_lua_block { local balancer = require "ngx.balancer" local host, port = my_load_backend_addr() assert(balancer.set_cur_peer(host, port)) } }

slide-17
SLIDE 17

ngx_stream_lua_module (TCP & UDP)

slide-18
SLIDE 18

Advanced debugging & profiling tools based on gdb & systemtap

slide-19
SLIDE 19

C2000K

slide-20
SLIDE 20

Web API and Microservices (real-time stock market info/weather)

slide-21
SLIDE 21

Web Gateways (HTTP/HTTPS/TCP/UDP, CDN)

slide-22
SLIDE 22

Web Applications (e-commerce/ad platforms/push systems/web sites)

slide-23
SLIDE 23
slide-24
SLIDE 24
slide-25
SLIDE 25

Distributed Storage Systems

slide-26
SLIDE 26

Datanet is coming Find @jaksprats on twitter

slide-27
SLIDE 27

Conflict-free Replicated Data Types CRDT

slide-28
SLIDE 28
slide-29
SLIDE 29

♡ The sregex regex engine

slide-30
SLIDE 30
slide-31
SLIDE 31
slide-32
SLIDE 32
slide-33
SLIDE 33
slide-34
SLIDE 34
slide-35
SLIDE 35
slide-36
SLIDE 36
slide-37
SLIDE 37
slide-38
SLIDE 38

♡ A web platform as a virtual machine

slide-39
SLIDE 39

lz.taobao.com

slide-40
SLIDE 40
slide-41
SLIDE 41
slide-42
SLIDE 42
slide-43
SLIDE 43
slide-44
SLIDE 44
slide-45
SLIDE 45

I invented the LZSQL language for the core business system.

slide-46
SLIDE 46

­­ /=/view/itemdailyflow/type/trend int $uid; text $begin, $end, $today, $url_index; symbol $db; location $lz_report; @hist := select ... from LZDB.dpunit_purl_result($db, $begin, $end, $uid) as a ... at $lz_report; @rt := select name, count(name) from LZRTI.getPurl($end as day, $uid) group by name ... return select ... from @hist union all @rt ...

slide-47
SLIDE 47
slide-48
SLIDE 48

$ lzsql­compile ­c ­O2 ­n src/*.lzsql $ lzsql­link ­m lightface.core \ ­o lightface/core.lua src/*.oul

slide-49
SLIDE 49
slide-50
SLIDE 50
slide-51
SLIDE 51

♡ The LZSQL compiler was written in just 4000 lines of Perl.

slide-52
SLIDE 52

LZSQL Parser LZSQL AST LZSQL Optimizers LZSQL Code Emitter

slide-53
SLIDE 53

♡ We can generate Lua code, so why not C?

slide-54
SLIDE 54

♡ The ngx_lz_st module was generated automatically from the real-time stats engine's TCP protocol documentation.

slide-55
SLIDE 55

♡ The documentation had been originally written in wiki format and then ported over to a declarative little language named Ticpy designed by myself.

slide-56
SLIDE 56

// File lzrti.tcp ... // fid=7 // @desc: obtain info for a specific shop and a specific visitor // @param uid: unit_id // @param uv: visitor cookie // @param all_sessions: whether return p4p hits: 1 is yes; 0 otherwise getUv(uid:4, uv:s, all_sessions:4): ( login_time:8, // session start time login_url:s, // session entry page login_title:s, // session entry page title stay_time:4, // current session elapsed time page_deep:4, // current session's access depth im:s, // user's IM account ip:s, // ip address location_id:4, // address id ... )

slide-57
SLIDE 57
slide-58
SLIDE 58

$ wc ­l lzrti.tcp protocol.c.tt ngx_http_lz_st.c 345 lzrti.tcp 464 protocol.c.tt 12478 ngx_http_lz_st.c 13287 total

slide-59
SLIDE 59

♡ So...I'd rather write programs to write programs to write programs...

slide-60
SLIDE 60

♡ Test our nginx C modules by

  • ur open source Perl module Test::Nginx!

http://search.cpan.org/perldoc?Test::Nginx

slide-61
SLIDE 61

use Test::Nginx::Socket 'no_plan'; run_tests(); __DATA__ === TEST 1: sanity ­­­ config location /main { echo_subrequest GET /sub; } location /sub { echo hello; } ­­­ request GET /main ­­­ response_body hello

slide-62
SLIDE 62

♡ Test the application web serives by Perl.

slide-63
SLIDE 63

use t::LZ; plan tests => 1 * blocks(); run_tests(); __DATA__ === TEST 1: lzers ­­­ url /=/view/lzers/~/~ ­­­ res [ { "cnt" : 911619 } ]

slide-64
SLIDE 64

=== TEST 5: sellerbasic ­­­ uid: 1509 ­­­ url /=/view/sellerbasic/~/~ ­­­ res [ { "uv_times" : 257 }, { "login_title" : "店内搜索页", "login_url" : "http://xiaowangpu...", ...

slide-65
SLIDE 65

♡ Generate test databases from rules by my open source Cheater utility written by Perl. http://search.cpan.org/perldoc?Cheater

slide-66
SLIDE 66
slide-67
SLIDE 67

♡ The OpenResty Model Language

slide-68
SLIDE 68

♡ The OpenResty View Language

slide-69
SLIDE 69

Jemplate & Lemplate

slide-70
SLIDE 70

♡ The OpenResty Controller Language

slide-71
SLIDE 71

uri­prefix("/foo/"), !uri­arg("_t") => redirect("/bar/"), done; uri­suffix(".jpg", ".gif", ".png") => exit(403);

slide-72
SLIDE 72

resp­filter { # skip C/C++ string literals: rx{ ' (?: \ [^\n] | [^'\n] )* ' } ­> $0; rx{ " (?: \ [^\n] | [^"\n] )* " } ­> $0; # remove all those ugly C/C++ comments: rx{ /\* .*? \*/ | // [^\n]* } ­> ''; }

slide-73
SLIDE 73

♡ WAF on the Controller Language.

slide-74
SLIDE 74

♡ ModSecurity is a horrible DSL.

slide-75
SLIDE 75 SecRule ARGS "(?i)([\s\"'`;\/0­9\=]+on\w+\s*=)" "id:'973337',phase:2,t:none,rev:'1',ver:'OWASP_CRS/2.2.9', maturity:'1',accuracy:'8',t:urlDecodeUni,t:htmlEntityDecode, t:jsDecode,t:cssDecode,log,capture, msg:'XSS Filter ­ Category 2: Event Handler Vector', tag:'OWASP_CRS/WEB_ATTACK/XSS',tag:'WASCTC/WASC­8', tag:'WASCTC/WASC­22',tag:'OWASP_TOP_10/A2', tag:'OWASP_AppSensor/IE1',tag:'PCI/6.5.1', logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: % {MATCHED_VAR}', severity:'2',setvar:'tx.msg=%{rule.msg}', setvar:tx.xss_score=+%{tx.critical_anomaly_score}, setvar:tx.anomaly_score=+%{tx.critical_anomaly_score}, setvar:tx.%{rule.id}­OWASP_CRS/WEB_ATTACK/XSS­%{matched_var_name}=%{tx.0}"
slide-76
SLIDE 76

☺ Model (Data analytics, Microservices) ☺ View (Web pages) ☺ Controller (CDN, WAF, Microservices)

slide-77
SLIDE 77

♡ SportLang

slide-78
SLIDE 78

♡ The Y Language gdb/python lldb/python systemtap bcc/ebpf luajit/ebpf

slide-79
SLIDE 79

♡ CoffeeScript

slide-80
SLIDE 80

♡ The meta domain specific language A DSL for creating all the DSLs (including itself).

slide-81
SLIDE 81

♡ Clean separation between business representation and business implementation

slide-82
SLIDE 82

♡ Compiling-style Web Frameworks

slide-83
SLIDE 83

♡ The best language is the business language.

slide-84
SLIDE 84

♡ Machine truly understands your business logic.

slide-85
SLIDE 85

☺ Any questions? ☺

slide-86
SLIDE 86