Strict validation versus accepting anything Evan Jones Bluecore - - PowerPoint PPT Presentation

▶
strict validation versus accepting anything
SMART_READER_LITE
LIVE PREVIEW

Strict validation versus accepting anything Evan Jones Bluecore - - PowerPoint PPT Presentation

Aug 19, 2023 147 likes •510 views

Strict validation versus accepting anything Evan Jones Bluecore Personalized e-commerce marketing ~4 years ~140 employees ~35 engineers Recommendations need product data 2. Page loads JS Partner-specific JS 3. User action sent Data

slide-1
SLIDE 1

Strict validation versus accepting anything

Evan Jones

slide-2
SLIDE 2

Bluecore

Personalized e-commerce marketing ~4 years ~140 employees ~35 engineers

slide-3
SLIDE 3
slide-4
SLIDE 4
slide-5
SLIDE 5

Recommendations need product data

slide-6
SLIDE 6
  • 1. User visits site
  • 2. Page loads JS
  • 3. User action sent

Partner-specific JS Data Ingestion DB Rules, Recommendations Email

  • 4. Find customers
  • 5. Send email
slide-7
SLIDE 7

Data ingestion

Web Handler Customer events (thousands/second) Queue Process Data Database

slide-8
SLIDE 8

Data ingestion

Web Handler Customer events (thousands/second) Queue Process Data Database

slide-9
SLIDE 9
slide-10
SLIDE 10
slide-11
SLIDE 11
slide-12
SLIDE 12
slide-13
SLIDE 13

Product data

"id": 429174, "name": "Pilot G2 Premium Retractable...", "price": 13.99,

slide-14
SLIDE 14

Product data

"id": 429174, "name": "Pilot G2 Premium Retractable...", "price": 13.99,

slide-15
SLIDE 15
slide-16
SLIDE 16
slide-17
SLIDE 17

Product data

"id": "429174", // may contain letters "name": "Pilot G2 Premium Retractable...", "price": 13.99,

slide-18
SLIDE 18

Product data

"id": "429174", // may contain letters "name": "Pilot G2 Premium Retractable...", "price": 13.99,

slide-19
SLIDE 19
slide-20
SLIDE 20
slide-21
SLIDE 21

Product data

"id": "429174", // may contain letters "name": "Pilot G2 Premium Retractable...", "price": "£13.99, // may contain currency

slide-22
SLIDE 22

What is valid product data?

slide-23
SLIDE 23

Design A: Accept anything! Design B: Strict validation!

slide-24
SLIDE 24

Design choice: Validation versus flexibility

Programming languages: static versus dynamic typing Databases: Strict versus flexible schemas (SQL vs NoSQL)

slide-25
SLIDE 25

Robustness Principle (Postel’s Law)

“Be liberal in what you accept, and conservative in what you send”

Advantage: implementations can interoperate (e.g. TCP) Disadvantage: bugs can become “standard” (e.g HTML)

slide-26
SLIDE 26

Original policy: Accept anything

Rationale: One chance to store the data; fix it later Implementation: Store any key/value pairs

slide-27
SLIDE 27

Fun ensues ...

price: 13 (integer), 13.99 (float), “13.99”, “£13.99” products without ids products with both “title” and “name”

slide-28
SLIDE 28

Evaluation

+ Store any e-commerce data + Fix any data bugs

slide-29
SLIDE 29

Evaluation

+ Store any e-commerce data + Fix any data bugs

  • Processing is much harder
  • Harder to test if we are sending the right data
slide-30
SLIDE 30

Raw data Validation Core System

slide-31
SLIDE 31

Raw data Validation Core System Valid Everything

slide-32
SLIDE 32

Raw data Validation Core System Valid One-off fix Everything

slide-33
SLIDE 33

Conclusion: Err on the side of validation

Find errors sooner Simplifies the overall system Easier to relax restrictions than to add them Want to fix errors later? Record everything

slide-34
SLIDE 34

Thanks!

Evan Jones http://www.evanjones.ca/ Bluecore http://www.bluecore.com/

slide-35
SLIDE 35

Store raw but require “core” schema

Store the raw data we receive Validate “core” fields: return helpful error messages e.g. must have id, price is a string, use “name” not “title” Found many data bugs