Star T EX: the Next Generation Why? Implementing T EX in Common - - PowerPoint PPT Presentation

▶

Dec 18, 2022 379 likes •555 views

Star T EX Didier Verna Introduction Star T EX: the Next Generation Why? Implementing T EX in Common Lisp Common Lisp Built-in paradigms Extensibility How? API Didier Verna Compatibility Conclusion didier@lrde.epita.fr @didierverna

SLIDE 1

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

Star T EX: the Next Generation

Implementing T EX in Common Lisp Didier Verna

didier@lrde.epita.fr @didierverna facebook/didier.verna http://www.lrde.epita.fr/˜didier

TUG 2012, July 16 – 18

1/21

SLIDE 2

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

T EX The [final] frontier. These are the voyages, Of a software enterprise. Its continuing mission: To explore new tokens, To seek out a new life, New forms of implementation. . .

2/21

SLIDE 3

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

Don Knuth @ TUG 2010

Why did you design T EX as a macro-expansion based system?

1 Wanted something simple to use for my secretary 2 Computational resources at the time were limited 1 Is T

EX simple to use, really?

2 Computational resources are not limited anymore

4/21

SLIDE 4

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

A better T EX?

What would that be?

T EX’s strength is in the quality of its typesetting, not in its programmatic interface. Keep the typesetting functionality but provide. . . A more modern and consistent API Real programming capabilities Still simple to use (at least for simple things) Extensibility / customizability Backward Compatibility

5/21

SLIDE 5

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

Alternatives

eval4tex, perlT EX, QAT EX, PyT EX, python, sT EXme, LuaT EX, iT

EX. . .

Wrap T EX in a programming language Wrap a programming language in T EX Writing macros in another language Getting rid of macros Synchronous dual-process (std redirection / file I/O) Multi-pass What about a fully integrated approach?

I know, NTS is dead. . .

6/21

SLIDE 6

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

Outline

1

Introduction

2

Why Common Lisp? Common Lisp Built-in paradigms Extensibility

3

How to do it? API Compatibility

4

Conclusion

7/21

SLIDE 7

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

Why Common Lisp?

A language that doesn’t get in the way

Old language (= obsolete, = mature and modern) ANSI standard (1994) ⇒ stable Industrial-scale general purpose language

◮ Multi-paradigm ◮ Highly optimizable ◮ Pletora of libraries

Scripting / extension language

◮ Highly dynamic ◮ Highly reflexive ◮ Easy to learn (no syntax) 9/21

SLIDE 8

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

Built-in paradigms

Free of charge

key/value interface: functions lambda-lists Packages: ASDF systems Namespaces: Common Lisp packages Interactive behavior: conditions and restarts Dumping: Lisp images (idea: user-level dumping) Performance:

◮ Interpretation / Compilation / JIT-Compilation ◮ Static typing ◮ And again, dumping

. . .

10/21

SLIDE 9

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

Extensibility / Customizability

Tweak at will

Reflection (introspection / intercession) Structural:

◮ Package internals (::) ◮ . . .

Behavioral:

◮ Reader-macros ◮ . . . 11/21

SLIDE 10

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

Objectives

Remember them?

✗ A more modern and consistent API ✔ Real programming capabilities ✔ Still simple to use (at least for simple things) ✔ Extensibility / customizability ✗ Backward Compatibility

13/21

SLIDE 11

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

A more modern and consistent API

Programmatic T EX primitives

Parameters ⇒ Lisp variables badness Quantities ⇒ Lisp objects (setf baselineskip #g(b :plus x :minus y)) Commands ⇒ Lisp functions (input file) (hbox material) (hbox material :to dim) (hbox material :spread dim) (hbox-to dim material) (hbox-spread dim material) The typesetting subset of T EX No \def, \relax and friends

14/21

SLIDE 12

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

Backward Compatibility

With good’old T EX

Implement traditional T EX on top of procedural T EX (part of) T EX’s digestive engine Provide a way to plug Lisp code in T EX files T EX macros written in Lisp or direct Lisp code

15/21

SLIDE 13

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

TiCL architecture

An overly simplified, extremely naive, totally wrong view

17/21

SLIDE 14

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

What does it take to embed Lisp in T EX?

Provided T EX is written in Lisp, that is :-)

( in−package :com. d v l s o f t . t i c l ) ( in−readtable :com. d v l s o f t . t i c l ) ; ; ; ; U t i l i t i e s ( defun 2+ ( number ) "Like 1+ but add 2 instead." (+ number 2)) ( defun 3+ ( number ) "Like 1+ but add 3 instead." (+ number 3)) ; ; ; ; String u t i l i t i e s ( defun hex−char−p ( char ) "Return T if CHAR is in 0123456789abcdef." (member char ’ ( # \ 0 #\1 #\2 #\3 #\4 #\5 #\6 #\7 #\8 #\9 #\a #\b #\ c #\d #\e #\ f ) ) ) ( defun eol−index ( string &optional ( s t a r t 0)) "Return the next end-of-line index in STRING from START." ( position #\ Newline string : s t a r t s t a r t ) ) ( defun t r a i l − i n d e x ( string &optional ( s t a r t 0) ( end ( or ( eol−index string s t a r t ) ( length string ) ) ) ) "Return the next trail index in STRING between START and END." ( loop : with t r a i l − i n d e x := n i l : f o r index : from (1− end ) : downto s t a r t : i f ( char= ( aref string index ) #\Space ) : do ( setq t r a i l − i n d e x index ) : else : return t r a i l − i n d e x : f i n a l l y ( return t r a i l − i n d e x ) ) ) ; ; ; ; TeX processing ( defstruct tex−string string ; the s t r i n g being processed length ; the length

f STRING

( index 0) ; the index

the next character to process in STRING t r a i l − i n d e x ; the index

the space t r a i l

n the

current l i n e in STRING eol−index ; the index

the next newline character in STRING eolp ; whether we have reached the end of the current l i n e ( state :N) ) ; the current state

f TeX

( defun tex−string ( string &aux ( length ( length string ) ) ( eol−index ( eol−index string ) ) ( t r a i l − i n d e x ( t r a i l − i n d e x string 0 ( or eol−index length ) ) ) ) "Return a new processing context for STRING." ( make−tex−string : string string : length length : t r a i l − i n d e x t r a i l − i n d e x : eol−index eol−index ) ) ( defun get−tex−line ( tex−string ) "Initiate the next line of TEX-STRING." ; ; #### PORTME: using with−slots on a s t r u c t . ( with−slots ( string length index t r a i l − i n d e x eol−index eolp ) tex−string ( unless (= index length ) ( setf eol−index ( eol−index string index ) t r a i l − i n d e x ( t r a i l − i n d e x string index ( or eol−index length ) ) ) ) ( setf eolp n i l ) ) ) ( defun skip−tex−line ( tex−string ) "Discard the rest of the current line in TEX-STRING." ( with−slots ( length index eol−index eolp ) tex−string ( i f eol−index ( setf index (1+ eol−index ) eolp t ) ( setf index length ) ) ) ) ( defvar endlinechar #\ Return "\\endlinechar. The character to insert at end of lines.") # i ( get−tex−char 1) ( defun get−tex−char ( tex−string &optional lookup ) "Get the next TeX character from TEX-STRING. Return either the next character on the current line, :EOL if the whole current line has been read, or nil if there is nothing left to read. If LOOKUP, don’t actually eat up the character in question." ; ; #### PORTME: using with−slots on a s t r u c t . ( with−slots ( string length index t r a i l − i n d e x eol−index eolp ) tex−string ( unless (= index length ) ; ; #### NOTE: even when we j u s t LOOKUP the next character without ; ; a c t u a l l y eating i t , i t doesn ’ t hurt to skip the t r a i l i n g spaces f o r ; ; real . (when ( eql index t r a i l − i n d e x ) ( setf index eol−index ) ) (when index (cond ( eolp : eol ) ( ( eql index eol−index ) ( prog1 endlinechar ( unless lookup ( incf index ) ( setf eolp t ) ) ) ) ((= ( catcode ( aref string index ) ) + superscript +) ( l e t ( ( char ( aref string index ) ) ( l i m i t ( or eol−index length ) ) ) (cond ( ( and (>= (− l i m i t index ) 4) ( char= char ( aref string (1+ index ) ) ) ( hex−char−p ( aref string (2+ index ) ) ) ( hex−char−p ( aref string (3+ index ) ) ) ) ( prog1 ( code−char ( read−from−string ( format n i l "#x~C~C" ( aref string (2+ index ) ) ( aref string (3+ index ) ) ) ) ) ( unless lookup ( incf index 4 ) ) ) ) ( ( and (>= (− l i m i t index ) 3) ( char= char ( aref string (1+ index ) ) ) (< ( char−code ( aref string (2+ index ) ) ) 128)) ( l e t ( ( char−code ( char−code ( aref string (2+ index ) ) ) ) ) ( prog1 ( code−char (+ char−code ( i f (< char−code 64) 64 −64))) ( unless lookup ( incf index 3 ) ) ) ) ) ( t ( prog1 ( aref string index ) ( unless lookup ( incf index ) ) ) ) ) ) ) ((= ( catcode ( aref string index ) ) +subscript +) ( l e t ( ( char ( aref string index ) ) ( l i m i t ( or eol−index length ) ) ) (cond ( ( and (>= (− l i m i t index ) 2) ( char= char ( aref string (1+ index ) ) ) ) ( multiple−value−bind ( object new−index ) ( read−from−string string t n i l : s t a r t (2+ index ) : preserve−whitespace t ) ( l e t ( ( new−string ( eval

bject ) ) )

( setf string ( concatenate ’ string (subseq string 0 index ) (when ( stringp new−string ) new−string ) (when (< new−index length ) (subseq string new−index ) ) ) length ( length string ) ) ) ( get−tex−line tex−string ) ( get−tex−char tex−string lookup ) ) ) ( t ( prog1 ( aref string index ) ( unless lookup ( incf index ) ) ) ) ) ) ) ( t ( prog1 ( aref string index ) ( unless lookup ( incf index ) ) ) ) ) ) ) ) ) ( defun process−tex−string ( tex−string ) "Process TEX-STRING." ( with−slots ( state ) tex−string ( loop : f o r char := ( get−tex−char tex−string ) : while char : i f (eq char : eol ) : do ( get−tex−line tex−string ) : and : do ( setf state :N) : else : do ( l e t ( ( catcode ( catcode char ) ) ) (cond ((= catcode +escape+) ( l e t

∗ ( ( char ( get−tex−char

tex−string ) ) ( catcode ( catcode char ) ) ) (cond ( ( eq char : eol ) ( eat ’ ) ( setf state :M) ) ( ( / = catcode + l e t t e r +) ( eat ( intern ( make−string 1 : initial−element char ) ) ) ( setf state ( i f (= catcode +space+) :S :M) ) ) ( t ( l e t ( ( name ( l i s t char ) ) ) ( loop : f o r char := ( get−tex−char tex−string : lookup ) : while (and char ( not (eq char : eol ) ) (= ( catcode char ) + l e t t e r +)) : do ( get−tex−char tex−string ) : do (push char name) : f i n a l l y ( eat ( intern ( coerce ( nreverse name) ’ string ) ) ) ) ( setf state :S ) ) ) ) ) ) ( (member catcode ‘( ,+ beginning−of−group+ ,+end−of−group+ ,+ math−shift+ ,+ alignment−tab+ ,+ parameter+ ,+ superscript+ ,+ subscript+ ,+ l e t t e r + ,+ other+ ,+ active +) : t e s t # ’=) ( eat char catcode ) ( setf state :M) ) ((= catcode +end−of−line +) ( skip−tex−line tex−string ) ( case state ( :N ( eat ’par ) ) ( :M ( eat #\Space +space +)) ( : S ) ) ) ((= catcode +ignored +)) ((= catcode +space+) (when (eq state :M) ( eat #\Space +space+) ( setf state :S ) ) ) ((= catcode +comment+) ( skip−tex−line tex−string ) ) ((= catcode + i n v a l i d +) (warn "Invalid character ’~C’ in input." char ) ) ) ) ) ) ) ( defun process−string ( string ) "Process STRING." ( process−tex−string ( tex−string string ) ) )

18/21

SLIDE 15

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

Expected problems

Let’s be realistic. . .

Huge task

◮ CFFI ◮ Compatibility mode

T EX’s digestive engine is not really a pipeline Lisp / traditional T EX interaction tricky Sandboxing Too much intercession. . . All the things I haven’t thought of yet (a lot)

19/21

SLIDE 16

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

These were the voyages, Of a software enterprise. Its continuing mission: To explore new tokens, To seek out a new life, New forms of implementation. To \textbf{go}, Where no T EX has gone before!

20/21

SLIDE 17

Star T EX Didier Verna Introduction Why?

Common Lisp Built-in paradigms Extensibility

How?

API Compatibility

Conclusion

Live long and prosper!

Questions?

21/21