Data-transformer library Michael Raskin Moscow Center for - - PowerPoint PPT Presentation

▶
data transformer library
SMART_READER_LITE
LIVE PREVIEW

Data-transformer library Michael Raskin Moscow Center for - - PowerPoint PPT Presentation

Apr 30, 2023 190 likes •655 views

Data-transformer library Michael Raskin Moscow Center for Continuous Mathematical Education April 2013 Michael Raskin (MCCME) Data-transformer library April 2013 1 / 16 User-visible problem: Input data is garbage Input data requirements are

slide-1
SLIDE 1

Data-transformer library

Michael Raskin

Moscow Center for Continuous Mathematical Education

April 2013

Michael Raskin (MCCME) Data-transformer library April 2013 1 / 16

slide-2
SLIDE 2

User-visible problem: Input data is garbage Input data requirements are constantly changing Programmer’s problems: How to define data format? How to keep all the code that touches a single piece of data always in sync?

Michael Raskin (MCCME) Data-transformer library April 2013 2 / 16

slide-3
SLIDE 3

User-visible problem: Input data is garbage Input data requirements are constantly changing Programmer’s problems: How to define data format? How to keep all the code that touches a single piece of data always in sync?

Michael Raskin (MCCME) Data-transformer library April 2013 2 / 16

slide-4
SLIDE 4

Spreadsheet: “555-01-55 uses 2-digit year format, corrected to 555-01-1955” User: “Let’s just sort the first column without sorting the second one then”

Michael Raskin (MCCME) Data-transformer library April 2013 3 / 16

slide-5
SLIDE 5

Spreadsheet: “555-01-55 uses 2-digit year format, corrected to 555-01-1955” User: “Let’s just sort the first column without sorting the second one then”

Michael Raskin (MCCME) Data-transformer library April 2013 3 / 16

slide-6
SLIDE 6

Initial scope: Validating CSV data Storing the data into a database Presenting data in a PDF Later added to scope: Web forms Currently web forms, csv files and SQL DB is used both for input and for

  • utput in production; PDF export is also used all the time

And validating logic keeps getting more complex. . .

Michael Raskin (MCCME) Data-transformer library April 2013 4 / 16

slide-7
SLIDE 7

Initial scope: Validating CSV data Storing the data into a database Presenting data in a PDF Later added to scope: Web forms Currently web forms, csv files and SQL DB is used both for input and for

  • utput in production; PDF export is also used all the time

And validating logic keeps getting more complex. . .

Michael Raskin (MCCME) Data-transformer library April 2013 4 / 16

slide-8
SLIDE 8

Initial scope: Validating CSV data Storing the data into a database Presenting data in a PDF Later added to scope: Web forms Currently web forms, csv files and SQL DB is used both for input and for

  • utput in production; PDF export is also used all the time

And validating logic keeps getting more complex. . .

Michael Raskin (MCCME) Data-transformer library April 2013 4 / 16

slide-9
SLIDE 9

Initial scope: Validating CSV data Storing the data into a database Presenting data in a PDF Later added to scope: Web forms Currently web forms, csv files and SQL DB is used both for input and for

  • utput in production; PDF export is also used all the time

And validating logic keeps getting more complex. . .

Michael Raskin (MCCME) Data-transformer library April 2013 4 / 16

slide-10
SLIDE 10

Initial scope: Validating CSV data Storing the data into a database Presenting data in a PDF Later added to scope: Web forms Currently web forms, csv files and SQL DB is used both for input and for

  • utput in production; PDF export is also used all the time

And validating logic keeps getting more complex. . .

Michael Raskin (MCCME) Data-transformer library April 2013 4 / 16

slide-11
SLIDE 11

Initial scope: Validating CSV data Storing the data into a database Presenting data in a PDF Later added to scope: Web forms Currently web forms, csv files and SQL DB is used both for input and for

  • utput in production; PDF export is also used all the time

And validating logic keeps getting more complex. . .

Michael Raskin (MCCME) Data-transformer library April 2013 4 / 16

slide-12
SLIDE 12

Portable schemas Portable Declarative Clear Restricted functionality Would my application survive porting anyway? (Spoiler: not likely) Make simple things declarative and complex things possible!

Michael Raskin (MCCME) Data-transformer library April 2013 5 / 16

slide-13
SLIDE 13

Portable schemas Portable Declarative Clear Restricted functionality Would my application survive porting anyway? (Spoiler: not likely) Make simple things declarative and complex things possible!

Michael Raskin (MCCME) Data-transformer library April 2013 5 / 16

slide-14
SLIDE 14

Portable schemas Portable Declarative Clear Restricted functionality Would my application survive porting anyway? (Spoiler: not likely) Make simple things declarative and complex things possible!

Michael Raskin (MCCME) Data-transformer library April 2013 5 / 16

slide-15
SLIDE 15

Portable schemas Portable Declarative Clear Restricted functionality Would my application survive porting anyway? (Spoiler: not likely) Make simple things declarative and complex things possible!

Michael Raskin (MCCME) Data-transformer library April 2013 5 / 16

slide-16
SLIDE 16

Classical procedural programming: there is code, and you pass it configuration and data Classical OOP: there is an object, which contains configuration, and data, and is associated with code Hybrid model: create an object keeping configuration and code, and briefly pass it actual data (usually used for complex processing of data streams)

Michael Raskin (MCCME) Data-transformer library April 2013 6 / 16

slide-17
SLIDE 17

Process one record at a time Schema: an s-expression, written by hand (can contain literal function values and what not) data-transformer instance: Stores the schema in a better format with some caches, hashes, etc. Briefly holds the data during processing Data is not stored, it gets loaded, processed and exported in a quick succession

Michael Raskin (MCCME) Data-transformer library April 2013 7 / 16

slide-18
SLIDE 18

Process one record at a time Schema: an s-expression, written by hand (can contain literal function values and what not) data-transformer instance: Stores the schema in a better format with some caches, hashes, etc. Briefly holds the data during processing Data is not stored, it gets loaded, processed and exported in a quick succession

Michael Raskin (MCCME) Data-transformer library April 2013 7 / 16

slide-19
SLIDE 19

Process one record at a time Schema: an s-expression, written by hand (can contain literal function values and what not) data-transformer instance: Stores the schema in a better format with some caches, hashes, etc. Briefly holds the data during processing Data is not stored, it gets loaded, processed and exported in a quick succession

Michael Raskin (MCCME) Data-transformer library April 2013 7 / 16

slide-20
SLIDE 20

Process one record at a time Schema: an s-expression, written by hand (can contain literal function values and what not) data-transformer instance: Stores the schema in a better format with some caches, hashes, etc. Briefly holds the data during processing Data is not stored, it gets loaded, processed and exported in a quick succession

Michael Raskin (MCCME) Data-transformer library April 2013 7 / 16

slide-21
SLIDE 21

Process one record at a time Schema: an s-expression, written by hand (can contain literal function values and what not) data-transformer instance: Stores the schema in a better format with some caches, hashes, etc. Briefly holds the data during processing Data is not stored, it gets loaded, processed and exported in a quick succession

Michael Raskin (MCCME) Data-transformer library April 2013 7 / 16

slide-22
SLIDE 22

Record is an array of fields Fields have parameters Specify them or use default values; default values can depend on the values of other fields Data is an array of values stored in the same order as field definitions

Michael Raskin (MCCME) Data-transformer library April 2013 8 / 16

slide-23
SLIDE 23

Record is an array of fields Fields have parameters Specify them or use default values; default values can depend on the values of other fields Data is an array of values stored in the same order as field definitions

Michael Raskin (MCCME) Data-transformer library April 2013 8 / 16

slide-24
SLIDE 24

Input verification Day, month and year should be numbers input string verification Year should be in the 20th or the 21th century field content verification Date should be possible, 31-02-2013 is a bad idea global (cross-field) verification

Michael Raskin (MCCME) Data-transformer library April 2013 9 / 16

slide-25
SLIDE 25

Input verification Day, month and year should be numbers input string verification Year should be in the 20th or the 21th century field content verification Date should be possible, 31-02-2013 is a bad idea global (cross-field) verification

Michael Raskin (MCCME) Data-transformer library April 2013 9 / 16

slide-26
SLIDE 26

Input verification Day, month and year should be numbers input string verification Year should be in the 20th or the 21th century field content verification Date should be possible, 31-02-2013 is a bad idea global (cross-field) verification

Michael Raskin (MCCME) Data-transformer library April 2013 9 / 16

slide-27
SLIDE 27

Input verification Day, month and year should be numbers input string verification Year should be in the 20th or the 21th century field content verification Date should be possible, 31-02-2013 is a bad idea global (cross-field) verification

Michael Raskin (MCCME) Data-transformer library April 2013 9 / 16

slide-28
SLIDE 28

(defparameter *basic-schema* ‘(((:code-name :captcha-answer) (:display-name "Task answer") (:type :int) (:string-verification-error "Please enter a number") (:data-verification-error "Wrong answer") (:string-export ,(constantly ""))) ((:code-name :email) (:display-name "Email") (:type :string) , (matcher "^(.+@.+[.].+|)\$") (:string-verification-error "Email is specified but it doesn’t look like a valid email address"))))

Michael Raskin (MCCME) Data-transformer library April 2013 10 / 16

slide-29
SLIDE 29

(let ((schema (transformer-schema-edit-field *basic-schema* :captcha-answer (lambda (x) (set-> x :data-verification (lambda (y) (and y (= y captcha-answer)))))))) ; some code using the schema )

Michael Raskin (MCCME) Data-transformer library April 2013 11 / 16

slide-30
SLIDE 30

Typical attributes: code name (for HTML form, SQL schemas, etc.) (:code-name :captcha-answer) readable name (:display-name "Task answer") field type (mainly for SQL schemas; also sets reasonable defaults for validation and parsing) (:type :int) validation procedures and error messages (:string-verification-error "Please enter a number") data formatting (:string-export ,(constantly ""))

Michael Raskin (MCCME) Data-transformer library April 2013 12 / 16

slide-31
SLIDE 31

Typical attributes: code name (for HTML form, SQL schemas, etc.) (:code-name :captcha-answer) readable name (:display-name "Task answer") field type (mainly for SQL schemas; also sets reasonable defaults for validation and parsing) (:type :int) validation procedures and error messages (:string-verification-error "Please enter a number") data formatting (:string-export ,(constantly ""))

Michael Raskin (MCCME) Data-transformer library April 2013 12 / 16

slide-32
SLIDE 32

Typical attributes: code name (for HTML form, SQL schemas, etc.) (:code-name :captcha-answer) readable name (:display-name "Task answer") field type (mainly for SQL schemas; also sets reasonable defaults for validation and parsing) (:type :int) validation procedures and error messages (:string-verification-error "Please enter a number") data formatting (:string-export ,(constantly ""))

Michael Raskin (MCCME) Data-transformer library April 2013 12 / 16

slide-33
SLIDE 33

Typical attributes: code name (for HTML form, SQL schemas, etc.) (:code-name :captcha-answer) readable name (:display-name "Task answer") field type (mainly for SQL schemas; also sets reasonable defaults for validation and parsing) (:type :int) validation procedures and error messages (:string-verification-error "Please enter a number") data formatting (:string-export ,(constantly ""))

Michael Raskin (MCCME) Data-transformer library April 2013 12 / 16

slide-34
SLIDE 34

Typical attributes: code name (for HTML form, SQL schemas, etc.) (:code-name :captcha-answer) readable name (:display-name "Task answer") field type (mainly for SQL schemas; also sets reasonable defaults for validation and parsing) (:type :int) validation procedures and error messages (:string-verification-error "Please enter a number") data formatting (:string-export ,(constantly ""))

Michael Raskin (MCCME) Data-transformer library April 2013 12 / 16

slide-35
SLIDE 35

CAPTCHA verification is injected into the schema right before use (transformer-schema-edit-field *basic-schema* :captcha-answer (lambda (x) (set-> x :data-verification (lambda (y) (and y (= y captcha-answer))))))

Michael Raskin (MCCME) Data-transformer library April 2013 13 / 16

slide-36
SLIDE 36

Channel-specific features CSV: convert date fields into triples of fields for data components (also for web forms) SQL: specifying foreign keys; WHERE-conditions and source tables for generating queries Web forms: HTTP POST requests; file upload handler Web forms and PDF: preparation for CL-Emb templates

Michael Raskin (MCCME) Data-transformer library April 2013 14 / 16

slide-37
SLIDE 37

Was it a good idea? Reinventing the wheel? No previous wheel found Very few cases of save and load (or similar) code being mismatched. Nice Complex checks are still simple to integrate The more advanced, the less portable Some schema field parameters are coupled not just to Common Lisp, but to CLSQL, CL-Emb, etc. Feature-poor portable declarative schemas are generated automatically and correctly when needed Schema-using code mostly untouched; individual schemas relatively short and simple. Helps near deadlines in understaffed projects Small API quirks accumulate. Not specific to our library Wasteful implementations of some functionality

Michael Raskin (MCCME) Data-transformer library April 2013 15 / 16

slide-38
SLIDE 38

Was it a good idea? Reinventing the wheel? No previous wheel found Very few cases of save and load (or similar) code being mismatched. Nice Complex checks are still simple to integrate The more advanced, the less portable Some schema field parameters are coupled not just to Common Lisp, but to CLSQL, CL-Emb, etc. Feature-poor portable declarative schemas are generated automatically and correctly when needed Schema-using code mostly untouched; individual schemas relatively short and simple. Helps near deadlines in understaffed projects Small API quirks accumulate. Not specific to our library Wasteful implementations of some functionality

Michael Raskin (MCCME) Data-transformer library April 2013 15 / 16

slide-39
SLIDE 39

Was it a good idea? Reinventing the wheel? No previous wheel found Very few cases of save and load (or similar) code being mismatched. Nice Complex checks are still simple to integrate The more advanced, the less portable Some schema field parameters are coupled not just to Common Lisp, but to CLSQL, CL-Emb, etc. Feature-poor portable declarative schemas are generated automatically and correctly when needed Schema-using code mostly untouched; individual schemas relatively short and simple. Helps near deadlines in understaffed projects Small API quirks accumulate. Not specific to our library Wasteful implementations of some functionality

Michael Raskin (MCCME) Data-transformer library April 2013 15 / 16

slide-40
SLIDE 40

Was it a good idea? Reinventing the wheel? No previous wheel found Very few cases of save and load (or similar) code being mismatched. Nice Complex checks are still simple to integrate The more advanced, the less portable Some schema field parameters are coupled not just to Common Lisp, but to CLSQL, CL-Emb, etc. Feature-poor portable declarative schemas are generated automatically and correctly when needed Schema-using code mostly untouched; individual schemas relatively short and simple. Helps near deadlines in understaffed projects Small API quirks accumulate. Not specific to our library Wasteful implementations of some functionality

Michael Raskin (MCCME) Data-transformer library April 2013 15 / 16

slide-41
SLIDE 41

Was it a good idea? Reinventing the wheel? No previous wheel found Very few cases of save and load (or similar) code being mismatched. Nice Complex checks are still simple to integrate The more advanced, the less portable Some schema field parameters are coupled not just to Common Lisp, but to CLSQL, CL-Emb, etc. Feature-poor portable declarative schemas are generated automatically and correctly when needed Schema-using code mostly untouched; individual schemas relatively short and simple. Helps near deadlines in understaffed projects Small API quirks accumulate. Not specific to our library Wasteful implementations of some functionality

Michael Raskin (MCCME) Data-transformer library April 2013 15 / 16

slide-42
SLIDE 42

Was it a good idea? Reinventing the wheel? No previous wheel found Very few cases of save and load (or similar) code being mismatched. Nice Complex checks are still simple to integrate The more advanced, the less portable Some schema field parameters are coupled not just to Common Lisp, but to CLSQL, CL-Emb, etc. Feature-poor portable declarative schemas are generated automatically and correctly when needed Schema-using code mostly untouched; individual schemas relatively short and simple. Helps near deadlines in understaffed projects Small API quirks accumulate. Not specific to our library Wasteful implementations of some functionality

Michael Raskin (MCCME) Data-transformer library April 2013 15 / 16

slide-43
SLIDE 43

Was it a good idea? Reinventing the wheel? No previous wheel found Very few cases of save and load (or similar) code being mismatched. Nice Complex checks are still simple to integrate The more advanced, the less portable Some schema field parameters are coupled not just to Common Lisp, but to CLSQL, CL-Emb, etc. Feature-poor portable declarative schemas are generated automatically and correctly when needed Schema-using code mostly untouched; individual schemas relatively short and simple. Helps near deadlines in understaffed projects Small API quirks accumulate. Not specific to our library Wasteful implementations of some functionality

Michael Raskin (MCCME) Data-transformer library April 2013 15 / 16

slide-44
SLIDE 44

Was it a good idea? Reinventing the wheel? No previous wheel found Very few cases of save and load (or similar) code being mismatched. Nice Complex checks are still simple to integrate The more advanced, the less portable Some schema field parameters are coupled not just to Common Lisp, but to CLSQL, CL-Emb, etc. Feature-poor portable declarative schemas are generated automatically and correctly when needed Schema-using code mostly untouched; individual schemas relatively short and simple. Helps near deadlines in understaffed projects Small API quirks accumulate. Not specific to our library Wasteful implementations of some functionality

Michael Raskin (MCCME) Data-transformer library April 2013 15 / 16

slide-45
SLIDE 45

Was it a good idea? Reinventing the wheel? No previous wheel found Very few cases of save and load (or similar) code being mismatched. Nice Complex checks are still simple to integrate The more advanced, the less portable Some schema field parameters are coupled not just to Common Lisp, but to CLSQL, CL-Emb, etc. Feature-poor portable declarative schemas are generated automatically and correctly when needed Schema-using code mostly untouched; individual schemas relatively short and simple. Helps near deadlines in understaffed projects Small API quirks accumulate. Not specific to our library Wasteful implementations of some functionality

Michael Raskin (MCCME) Data-transformer library April 2013 15 / 16

slide-46
SLIDE 46

Thanks for your attention!

Michael Raskin (MCCME) Data-transformer library April 2013 16 / 16