Development of SML# making ML an ordinary practical languaage . . - - PowerPoint PPT Presentation

. . . . . . .

Development of SML# – making ML an ordinary practical languaage

Atsushi Ohori joint work with Katsuhiro Ueno

, Tohoku University

Atsushi Ohori Development of SML# – making ML an ordinary practical language 1 / 59

Motivation and backgrounds

This talk is about SML#

Its motivation:

to make ML an ordinary practical language.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59

Motivation and backgrounds

This talk is about SML#

Its motivation:

to make ML an ordinary practical language.

Some immediate reactions from eminent ML/FP researchers would be:

Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59

Motivation and backgrounds

This talk is about SML#

Its motivation:

to make ML an ordinary practical language.

Some immediate reactions from eminent ML/FP researchers would be: We should be taking about the glorious future of ML or other functional languages ...

Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59

Motivation and backgrounds

This talk is about SML#

Its motivation:

to make ML an ordinary practical language.

Some immediate reactions from eminent ML/FP researchers would be: We should be taking about the glorious future of ML or other functional languages ... ML has already been a super-advanced and highly practical language.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59

Motivation and backgrounds

This talk is about SML#

Its motivation:

to make ML an ordinary practical language.

Some immediate reactions from eminent ML/FP researchers would be: We should be taking about the glorious future of ML or other functional languages ... ML has already been a super-advanced and highly practical language. What’s the issue?

Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59

Motivation and backgrounds

Backgrounds

ML is indeed a great language. it is:

Highly Productive

We all know it.

Highly Reliable, and

Polymorphic type inference is the only verification tool made into practice for programmers’ daily use.

Safe

It is type safe and memory safe; “buffer overflow” is not even an issue. You get all of them for free by just using ML ....

Atsushi Ohori Development of SML# – making ML an ordinary practical language 3 / 59

Motivation and backgrounds

Backgrounds

ML is indeed a great language. it is:

Highly Productive

We all know it.

Highly Reliable, and

Polymorphic type inference is the only verification tool made into practice for programmers’ daily use.

Safe

It is type safe and memory safe; “buffer overflow” is not even an issue. You get all of them for free by just using ML ....

Then, everyone should have been using ML by now,...

Atsushi Ohori Development of SML# – making ML an ordinary practical language 3 / 59

Motivation and backgrounds Weaknesses of ML

Weaknesses of ML

Despite these advantages, ML has not yet been among the popular production languages of the industry. There may be historical or cultural backgrounds/prejudice. It is a toy invented in academia... Functional language is difficult/inefficient/impractical... But, more fundamentally, several important practical problems have been left unsolved. improper treatment of records lack of interoperability with C lack of database support lack of native thread support lack of separate compilation and linking lack of tools · · ·

Atsushi Ohori Development of SML# – making ML an ordinary practical language 4 / 59

Motivation and backgrounds Weaknesses of ML

Weakness (1)： improper treatment of records

Record is the most basic data constructor, which must be fully supported. However, record operations are not quite first-class citizens in ML. In SML:

• fn x => map #name x;

stdIn:1.1-1.17 Error: unresolved flex record (can’t tell what fields there are besides #name)

This is actually better than some of the other systems.

A minor comment: there are several solutions. I nonetheless address this issue first, since our solution to this problem turns out to be an important step in the development of SML#.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 5 / 59

Motivation and backgrounds Weaknesses of ML

Weakness (2) : lack of interoperability with C

Seamless (or at least smooth) interoperability with C is a prerequisite for a new language to be accepted in practice. Unfortunately, in current ML, using C libraries requires: to understand runtime representations of ML objects, to write a low-level stub for data conversion, and perhaps to call mysterious coordination-functions for GC which make using C not only cumbersome but also dangerous.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 6 / 59

Motivation and backgrounds Weaknesses of ML

Weakness (3) : lack of database support

There are few real-world applications that do not use databases. The current (ordinary) practice is string interface to SQL, where the programmer generates an SQL command string, sends it to an SQL server, and converts the result to appropriate data structure. This is cumbersome, error prone and unsafe. ML is worse: programmers cannot even use string interface to SQL.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 7 / 59

Motivation and backgrounds Weaknesses of ML

Weakness (4) : lack of native thread support

Exploiting emerging multi/many-core CPUs is the key to future high-performance application development. In the excising typical ML compilers, threads are implemented by their runtime system using a timer. This implies that only one core can be used, no matter how many threads the program code generates.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 8 / 59

Motivation and backgrounds Weaknesses of ML

Weakness (5) : lack of separate compilation

In a large software project, separate compilation and linking is essential. Unfortunately, no ML compiler supports separate compilation and linking in the ordinary sense, i.e. the process: to compile a source file into an object file consisting of

a set of machine code blocks external name definitions of the exported variables external name refereces of the imported variables

and to link an object file with

• ther ML object files independenly compiled, and

system libraries (libc, libm etc)

There seem to be no theoretical foundation for separately compiling Standard ML (except for typecheking.)

Atsushi Ohori Development of SML# – making ML an ordinary practical language 9 / 59

Motivation and backgrounds Sources of the weakness

Probable sources of these weakness

Widely accepted assumptions/beliefs/folklore around ML: Parametric polymorphism is the principle. Copying collector is the choice. Interfaces are Types Taking them as axioms, ML attempts to re-construct the world. This attempt may work well in theory, but creates some serious problems in practice. Let us examine their implication and impact.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 10 / 59

Motivation and backgrounds Sources of the weakness

Parametric Polymorphism Principle

Since ’a in ’a -> ’a ranges over any types, we should only have one representation, namely 32/64 bit word (possibly with an inline tag).

Atsushi Ohori Development of SML# – making ML an ordinary practical language 11 / 59

Motivation and backgrounds Sources of the weakness

Parametric Polymorphism Principle

Since ’a in ’a -> ’a ranges over any types, we should only have one representation, namely 32/64 bit word (possibly with an inline tag). In the ordinary daily computation (on any hardware), however, we have:

Atsushi Ohori Development of SML# – making ML an ordinary practical language 11 / 59

Motivation and backgrounds Sources of the weakness

Parametric Polymorphism Principle

Since ’a in ’a -> ’a ranges over any types, we should only have one representation, namely 32/64 bit word (possibly with an inline tag). In the ordinary daily computation (on any hardware), however, we have: Some programmers would surely give up polymorphism for these data.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 11 / 59

Motivation and backgrounds Sources of the weakness

Copy GC Folklore

Functional programs require fast allocation (true, indeed), on which folklore has it that: “Cheney’s copying GC is the only option for primary (minor) GC”. This implies that there is one allocation pointer for the thread-shared global heap, and

• bjects are moved at any unpredictable time.

In such an absurd environment, one reasonable strategy for the ML runtime is to act as a monitor for thread scheduling and memory allocation.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 12 / 59

Motivation and backgrounds Sources of the weakness

Copy GC Folklore

Functional programs require fast allocation (true, indeed), on which folklore has it that: “Cheney’s copying GC is the only option for primary (minor) GC”. This implies that there is one allocation pointer for the thread-shared global heap, and

• bjects are moved at any unpredictable time.

In such an absurd environment, one reasonable strategy for the ML runtime is to act as a monitor for thread scheduling and memory allocation. However, if we are free from the folklore, then there is a possibility of: having a non-moving GC with multiple allocation pointers, and directly using POSIX threads provided by the OS kernel. Then you get true efficient multi-core threads for free. This is the way it should be; OS kernels are designed to serve us.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 12 / 59

Motivation and backgrounds Sources of the weakness

Interface as Types Assumption

There are some proposals for (a theory of?) “separate compilation” system for ML, but as far as I can tell they only talk about type-checking, and do not address separate compilation.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 13 / 59

Motivation and backgrounds Sources of the weakness

Interface as Types Assumption

There are some proposals for (a theory of?) “separate compilation” system for ML, but as far as I can tell they only talk about type-checking, and do not address separate compilation. In the ordinary understanding, compilation is the process to translate a source program into an object file consisting of machine code blocks, a set of defined symbols for the set of exported variables, and a set of external symbols (to be resolved by a linker). Separate compilation is a process to compile (in the above sense) a source without consulting any source programs that they are referenced. The reslulting object files are linked by a system linker (ld) with

• ther object files separately compiled, and

system libraries (libc, libm etc). In this ordinary sense, there is no system that can separately compile Standard ML (other than SML#.)

Atsushi Ohori Development of SML# – making ML an ordinary practical language 13 / 59

Motivation and backgrounds Sources of the weakness

Interface as Types Assumption II

For example, consider compiling the following into linkable machine code:

functor F (type foo datatype bar = A of foo * int) = struct datatype hoge = X of int * X.bar fun f (a, b) = X (a, X.A (b, a)) fun g (X (_, X.A c)) = c end

It is rather trivial to do the following: .

.

.

1

separately typecheck sources and keep them (or elaboration of them) .

.

.

2

when all the programs become available, then do code generation from the sources (or their elaborated intermediate forms). but I would not consider such process as separate compilation. Blurring separate compilation with typecheking would curtail the

• pportunity for making ML an ordinary practical language.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 14 / 59

Motivation and backgrounds The SML# Project

My comments on the weakness of ML

Of course researchers have been aware that

• ne-bit tag in integers causes problems

heap-allocating floating point numbers is undesirable threads should be supported by the OS thread library, · · · On this situation, two comments I would like to make: .

.

.

1

They are ingenious mechanisms researchers have developed to realize ML, to which my thorough respects are due. .

.

.

2

Overcoming them has been shown to be quite difficult. They are the prices to pay for ML to obtain its reliability and productivity.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 15 / 59

Motivation and backgrounds The SML# Project

My comments on the weakness of ML

Of course researchers have been aware that

• ne-bit tag in integers causes problems

heap-allocating floating point numbers is undesirable threads should be supported by the OS thread library, · · · On this situation, two comments I would like to make: .

.

.

1

They are ingenious mechanisms researchers have developed to realize ML, to which my thorough respects are due. .

.

.

2

Overcoming them has been shown to be quite difficult. They are the prices to pay for ML to obtain its reliability and productivity. Unfortunately, these prices are too high to make ML an ordinary practical language.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 15 / 59

Motivation and backgrounds The SML# Project

My comments on the weakness of ML

Of course researchers have been aware that

• ne-bit tag in integers causes problems

heap-allocating floating point numbers is undesirable threads should be supported by the OS thread library, · · · On this situation, two comments I would like to make: .

.

.

1

They are ingenious mechanisms researchers have developed to realize ML, to which my thorough respects are due. .

.

.

2

Overcoming them has been shown to be quite difficult. They are the prices to pay for ML to obtain its reliability and productivity. Unfortunately, these prices are too high to make ML an ordinary practical language. As researchers and hackers∗ who love ML, we cannot stop here!

(*) A person with an enthusiasm for programming or using computers as an end in itself.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 15 / 59

Motivation and backgrounds The SML# Project

We must develop a new ML language that can be a possible choice of the industry for serious software development.

The SML# Project

Its goal is to develop a new language in the ML family that is: a proper extension of Standard ML:

record polymorphism (and others, e.g rank-1 polymorphism, and limited form of overloading) complete downward compatibility with Standard ML

with practically important features:

seamless interoperability with C seamless database integration separate compilation and linking direct and light-weight native thread support easy-to-use parallelism (being developed)

Atsushi Ohori Development of SML# – making ML an ordinary practical language 16 / 59

Motivation and backgrounds The SML# Project

SML# team and collaborators

SML# Development Team: Atsushi Ohori (RIEC, Tohoku University) Katsuhiro Ueno (RIEC, Tohoku University) in (past) collaboration with (current affiliation): Kiyoshi Yamatodani (Sanpu Kobo Inc.) Nguen Duc Huu (Hanoi University of Science and Technology) Liu Bochao (CNCERT, China) Satoshi Osaka (Advantest) with contributions from many peoples including: Nobuaki Yoshida, Isao Sasano, Yutaka Matsuno, Kwanghoon Choi

Atsushi Ohori Development of SML# – making ML an ordinary practical language 17 / 59

Quick overview of SML# through examples Record polymorphism

Rest of the Talk

. . .

1

Motivation and backgrounds . . .

2

Quick overview of SML# through examples Record polymorphism Interoperability with C Seamless database integration Separate compilation . . .

3

Outline of the Technical Developments

Atsushi Ohori Development of SML# – making ML an ordinary practical language 18 / 59

Quick overview of SML# through examples Record polymorphism

Record polymorphism: basic record operations

fn x => map #X x; val it = fn : [’a#{X:’b},’b.’a list -> ’b list] # fun getX x = #X x; val getX = fn : [’a#{X:’b},’b.’a -> ’b] # fun setX x y= x # {X = y}; val setX = fn : [’a#{X:’b},’b.’a -> ’b -> ’a]

Atsushi Ohori Development of SML# – making ML an ordinary practical language 19 / 59

Quick overview of SML# through examples Record polymorphism

Record polymorphism: objects

val methods = { getX = fn self => #X (!self), setX = fn S => fn x => S := (!S # {X = x}), getY = fn S => #Y (!S), setY = fn S => fn x => S := (!S # {Y = x}), getColor = fn S => #Color (!S), setColor = fn S => fn x => S := (!S # {Color = x}) }; local val state = ref { X = 0.0, Y = 0.0 } in val p1 = fn method => method methods state end; p1 # setX 1.0; p1 # getX;

Atsushi Ohori Development of SML# – making ML an ordinary practical language 20 / 59

Quick overview of SML# through examples Record polymorphism

Record polymorphism: polymorphic variants

val P1 = fn M => #CPoint M {x=10.0, y = 10.0}; val P2 = fn M => #PPoint M {r=14.1421356, theta = 45.0}; val myPointList = [P1, P2]; val distance = {CPoint = fn {x,y,...} => Real.Math.sqrt (x *x + y* y), PPoint = fn {r, theta,...} => r}; P1 distance ; P2 distance ; fun iterator L f = map (fn x => x f) L; val myPointLstIter = fn x => iterator myPointList x; myPointLstIter distance;

See (Ohori, 1995) for formal accounts of polymorphic variants.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 21 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (1): the “Hello World”

SML# source program hello.sml:

val print = _import "printf" : string -> int val _ = print "こんにちは，日本へようこそ．\n"

Compile, link and execution:

% smlsharp hello.sml % ./a.out こんにちは，日本へようこそ．

The above “printf” is the printf in libc library, of course.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 22 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (2): printf

Source program printf.sml:

val printReal = _import "printf" : (string, real) -> int val printTriple = _import "printf" : (string, real, real, real) -> int val _ = printReal("%f\n", 3.14) val _ = printTriple("if a=%f, b=%f then a/b=%f\n, 3.14, 1.41, 3.14/1.41)

Compile, link and execute:

% smlsharp printf.sml % ./a.out 3.140000 if a=3.140000, b=1.410000 then a/b=2.226950

Atsushi Ohori Development of SML# – making ML an ordinary practical language 23 / 59

Quick overview of SML# through examples Interoperability with C

Some comment on “printf”

There are some proposals for printf in ML. They are elegant, entertaining, and educational. They also roughly works as an alternative to printf, but not quite.

val printYen = _import "printf" : (string,int) -> int val _ = printYen("%dYen\n", 1024) val _ = printYen("%10dYen\n", 1024) val _ = printYen("%-10dYen\n", 1024) val _ = printYen("%+10dYen\n", 1024) val _ = printYen("%#10xYen\n", 1024) ...

and many more alternatives.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 24 / 59

Quick overview of SML# through examples Interoperability with C

Some comment on “printf”

There are some proposals for printf in ML. They are elegant, entertaining, and educational. They also roughly works as an alternative to printf, but not quite.

val printYen = _import "printf" : (string,int) -> int val _ = printYen("%dYen\n", 1024) val _ = printYen("%10dYen\n", 1024) val _ = printYen("%-10dYen\n", 1024) val _ = printYen("%+10dYen\n", 1024) val _ = printYen("%#10xYen\n", 1024) ...

and many more alternatives.

Remember what you have wanted: it’s printf. Then use printf.

It’s free and fully functional (of course since it is printf).

Atsushi Ohori Development of SML# – making ML an ordinary practical language 24 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (3): a small real example

mt.sml using Mersenne Twister: val printWord = _import "printf" : (string, word) -> int val printReal = _import "printf" : (string, real) -> int val init = _import "init_genrand" : (int) -> unit val genWord = _import "genrand_int32" : () -> word val genReal = _import "genrand_res53" : () -> real val _ = init 0 val _ = printWord ("%d\n", genWord ()) val _ = printReal ("%lf\n",genReal ()) % gcc -c mt19937ar.c -o mtlib.o # compile Mersenne Twister % smlsharp mt.sml mtlib.o % a.out 2357136044 0.592845

Atsushi Ohori Development of SML# – making ML an ordinary practical language 25 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (4): higher-order functions I

(Just for your entertainment) In SML#:

val f1 = _import "f1" : (((((()->unit)->unit)->unit)->unit)->unit)->unit val g1 = _import "g1" : ()->()->()->()->unit val () = f1 (fn h1 => (print "h1\n"; h1 (fn h2 =>(print "h2\n"; h2 (fn h3 => print "h3\n"))))) val g2 = g1 () val g3 = g2 () val g4 = g3 () val () = g4 ()

Atsushi Ohori Development of SML# – making ML an ordinary practical language 26 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (4): higher-order functions II

In C:

void f4(void){ printf("f4\n");} void f3(void(*f)(void(*)(void))) {printf("f3\n"); f(f4); } void f2(void(*f)(void(*)(void(*)(void(*)(void))))) {printf("f2\n");f(f3); } void f1(void(*f) (void(*)(void(*)(void(*)(void(*)(void(*)(void))))))) {printf("f1\n");f(f2);}

Atsushi Ohori Development of SML# – making ML an ordinary practical language 27 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (4): higher-order functions III

void g4(void) {printf("g4\n");} void (*g3(void))(void) {printf("g3\n");return g4;} void (*(*g2(void))(void))(void) {printf("g2\n");return g3;} void (*(*(*g1(void))(void))(void))(void) {printf("g1\n");return g2;}

Atsushi Ohori Development of SML# – making ML an ordinary practical language 28 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (4): higher-order functions IV

Compile, link and execute:

% smlsharp callback.sml callback.o % gcc -c callback.c % smlsharp callback.sml callback.o % ./a.out f1 h1 f2 h2 f3 h3 g1 g2 g3 g4

Atsushi Ohori Development of SML# – making ML an ordinary practical language 29 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (5): polymorphic functions

(For your further entertainment)

val libc = DynamicLink.dlopen libc val c_qsort = DynamicLink.dlsym (libc, "qsort") fun qsort (a, f) = _ffiapply c_qsort (a : ’a array, Array.length a : int, _sizeof(’a), f : (’a ptr, ’a ptr) -> int) : unit

In the interpreter mode:

# use "qsort.sml"; ... val qsort = fn : [’a. ’a array * (’a ptr * ’a ptr -> int) -> unit]

Atsushi Ohori Development of SML# – making ML an ordinary practical language 30 / 59

Quick overview of SML# through examples Seamless database integration

Seamless database integration in SML#

SML# achieves seamless databases integration where SQL expressions are first-class citizens in the language, and they are evaluated by a database server. As an example, consider the following SQL command:

Q = SELECT name, age FROM people WHERE age <= 25

which is a shorthand for the following verbose syntax:

Q = SELECT person.name AS name, person.age AS age FROM people AS person WHERE (person.age <= 25)

Atsushi Ohori Development of SML# – making ML an ordinary practical language 31 / 59

Quick overview of SML# through examples Seamless database integration

Simple demo: seamless database integration

# val Q = _sql db => select #person.name as name, #person.age as age from #db.people as person where SQL.<= (#person.age, 25); val q = fn : [’a#{people:’b}, ’b#{age:int, name:’d}, ’c, ’d::{int, word, char, string, real, ’e option}, ’e::{int, word, char, bool, string, real}. (’a, ’c) SQL.db -> {age: int, name: ’d} SQL.query]

Atsushi Ohori Development of SML# – making ML an ordinary practical language 32 / 59

Quick overview of SML# through examples Seamless database integration

Simple demo: seamless database integration

# val Q = _sql db => select #person.name as name, #person.age as age from #db.people as person where SQL.<= (#person.age, 25); val q = fn : [’a#{people:’b}, ’b#{age:int, name:’d}, ’c, ’d::{int, word, char, string, real, ’e option}, ’e::{int, word, char, bool, string, real}. (’a, ’c) SQL.db -> {age: int, name: ’d} SQL.query] Of course, you get a lot of type errors when you first write a complicated query.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 32 / 59

Quick overview of SML# through examples Seamless database integration

Simple demo: seamless database integration

# val server = _sqlserver "host=127.0.0.1 dbname=test" : {people: {name: string, age: int}}; val server = "host=127.0.0.1 dbname=test" : {people: {age: int, name: string}} SQL.server # val db = SQL.connect server; val db = <conn> : {people:{age: int, name: string}} SQL.conn # val r = _sqleval q db; val r = <rel> : {age: int, name: string} SQL.rel # val x = SQL.fetchAll r; val x = [{age = 25, name = "Bob"}] : {age: int, name: string} list

Atsushi Ohori Development of SML# – making ML an ordinary practical language 33 / 59

Quick overview of SML# through examples Separate compilation

Separate compilation in SML#

SML# compiles a sequence of any of type and datatype bindings variable and exception bindings structure bindings functor bindings into an object code in the standard ELF format, and links it with SML# runtime library, C libraries (linbc.a, libm.a, libgmp.a, etc), and user compiled C functions by invoking the system linker (ld) to produce an executable file.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 34 / 59

Quick overview of SML# through examples Separate compilation

SML# compiler command

Some typical usages:

smlsharp -c foo.sml

compiles foo.sml to an object file foo.o, consulting with its interface file specified implicitly or explicitly.

smlsharp -c foo.smi -o foo

locates the object files referenced in foo.smi, invokes the system linker to produce an executable file foo.

smlsharp foo.sml

compiles foo.sml to foo.o, links it with all the files referenced in foo.smi and generates a.out. An interface file foo.smi of foo.sml specifies: the resources required by foo.sml and, those foo.sml provides.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 35 / 59

Quick overview of SML# through examples Separate compilation

Separate compilation example I

top.sml: structure B = F(type foo = real datatype bar = A of foo * int) val printPair = _import "printPair" : real * int -> unit val _ = printPair (B.g (B.f (2011, 3.14))) top.smi: _require "functor.smi", "printPair.smi"

Atsushi Ohori Development of SML# – making ML an ordinary practical language 36 / 59

Quick overview of SML# through examples Separate compilation

Separate compilation example II

functor.smi: functor F(type foo; datatype bar = A of foo * int) = struct datatype hoge = X of int * X.bar val f : int * X.foo -> hoge val g : hoge -> X.foo * int end functor.sml: functor F (type foo datatype bar = A of foo * int) = struct datatype hoge = X of int * X.bar fun f (a, b) = X (a, X.A (b, a)) fun g (X (_, X.A c)) = c end

Atsushi Ohori Development of SML# – making ML an ordinary practical language 37 / 59

Quick overview of SML# through examples Separate compilation

Separate compilation example III

printPair.sml: val printPair = _import "c_printPair" : (real,int) -> int printPair.smi val printPair : real * int -> int

C file

struct PAIR {double a; int b;}; void c_printPair(struct PAIR *X) { printf ("(%4.2lf,%4ld)\n", X->a, X->b);}

Atsushi Ohori Development of SML# – making ML an ordinary practical language 38 / 59

Quick overview of SML# through examples Separate compilation

Separate compilation example IV

Compile, link and execute:

% smlsharp -c top.sml % smlsharp -c functor.sml % smlsharp -c printPair.sml % gcc -c printPair.c % smlahrp top.smi printPaur.o % a.out (3.14, 2011)

Atsushi Ohori Development of SML# – making ML an ordinary practical language 39 / 59

Outline of the Technical Developments

Contents of the Talk

. . .

1

Motivation and backgrounds . . .

2

Quick overview of SML# through examples . . .

3

Outline of the Technical Developments Polymorphic Record Calculus Natural data representation Separate compilation Seamless database integration Efficient Non-moving GC

Atsushi Ohori Development of SML# – making ML an ordinary practical language 40 / 59

Outline of the Technical Developments

Technical Development of SML#

SML# is realized by combining the following .

.

.

1

Polymorphic record calculus .

.

.

2

Natural data representation .

.

.

3

Separate compilation .

.

.

4

Polymorphic type system for databases .

.

.

5

Efficient non-moving GC

Atsushi Ohori Development of SML# – making ML an ordinary practical language 41 / 59

Outline of the Technical Developments Polymorphic Record Calculus

The key technique: polymorphic record compilation

The key insight underlying the development of SML# compiler is “Polymorphism” is just an abstraction, and not a computational reality. So it must be compiled out. The first example is the polymorphic record calculus and its compilation (Ohori 1992, 1995). Consider the term:

fn x => #name x

This is apparently polymorphic, but it is also apparent that this must be compiled out to efficient load instruction.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 42 / 59

Outline of the Technical Developments Polymorphic Record Calculus

Compiling polymorphic operation using types

First, we reconstruct a typed term:

val f : [’a,’b#{name:’a}. ’a -> ’b] = [’a, ’b#{name:’a}. fn )x:’b) => #name x] f {name:string, age:int} {name = "smlsharp", age=7}

where ’b#{name:’a} is a type variable with a record kind. We then insert abstraction and application to pass necessary attributes for compilation.

val f : [’a,’b#{name:’a}. index(’b,name) -> ’b -> ’a] = [’a, ’b#{name:’a}. fn (I:index(’b,name)) => fn (x:’b) => x[I]] f {name:string, age:int} 2 [| 7,"smlsharp" |]

Atsushi Ohori Development of SML# – making ML an ordinary practical language 43 / 59

Outline of the Technical Developments Polymorphic Record Calculus

Review of record compilation

It is based on the following ideas: .

.

.

1

Information needed to compile a polymorphic function is abstracted at type abstraction and applied at type application. .

.

.

2

The needed information is encoded as a type that denotes the unique needed value. In the example, index(τ, name) denotes the index value of the name field of a record type τ. This mechanism is far-reaching in compiling out various aspects of polymorphism, which we have used to develop SML# throughout.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 44 / 59

Outline of the Technical Developments Polymorphic Record Calculus

Review of record compilation

It is based on the following ideas: .

.

.

1

Information needed to compile a polymorphic function is abstracted at type abstraction and applied at type application. .

.

.

2

The needed information is encoded as a type that denotes the unique needed value. In the example, index(τ, name) denotes the index value of the name field of a record type τ. This mechanism is far-reaching in compiling out various aspects of polymorphism, which we have used to develop SML# throughout.

Note: These concepts have later been variously called such as “dictionary passing” “evidences” “singleton types” Although literature did not seem to properly compare or even mention, the essence is the same as the one first presented in the 1992 article.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 44 / 59

Outline of the Technical Developments Natural data representation

Natural data representation

In SML#, we have:

Atsushi Ohori Development of SML# – making ML an ordinary practical language 45 / 59

Outline of the Technical Developments Natural data representation

Natural data representation

In SML#, we have:

This is obviously desired. How the pointer-tracing GC works?

Atsushi Ohori Development of SML# – making ML an ordinary practical language 45 / 59

Outline of the Technical Developments Natural data representation

Natural data representation: object bitmap

We make the following heap blocks.

• th bit

1

non- pointer pointer

• th word
• bject pointer

header

(including size )

words layout bitmap bits

Note: This is only for records (tuples); atomic data (except for strings) are always unboxed.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 46 / 59

Outline of the Technical Developments Natural data representation

Natural data representation: object bitmap

We make the following heap blocks.

• th bit

1

non- pointer pointer

• th word
• bject pointer

header

(including size )

words layout bitmap bits

Note: This is only for records (tuples); atomic data (except for strings) are always unboxed. How such a bit map can be computed for polymorphic functions?

Atsushi Ohori Development of SML# – making ML an ordinary practical language 46 / 59

Outline of the Technical Developments Natural data representation

Natural data representation: object bitmap

We make the following heap blocks.

• th bit

1

non- pointer pointer

• th word
• bject pointer

header

(including size )

words layout bitmap bits

Note: This is only for records (tuples); atomic data (except for strings) are always unboxed. How such a bit map can be computed for polymorphic functions? We adopt the idea of record compilation.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 46 / 59

Outline of the Technical Developments Natural data representation

Natural data representation: compilation

We introduce the types

tag(τ) to denote one bit indicating whether τ is boxed or not size(τ) to denote the size of τ

The rest is essentially the same as record compilation, but requires certain amount of careful development, especially in closure conversion that creates records which require bitmaps, a call frame that also requires a bitmap, properly aligning record fields. See (Nguen Duc Huu and Atsushi Ohori, 2006) for the details.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 47 / 59

Outline of the Technical Developments Separate compilation

Separate compilation

I’d repeat: to make ML an ordinary practical language, it is essential to have separate compilation and linking in the ordinary sense, which is the process to compile a source file into an object file consisting of

a set of machine code blocks, external name definitions of the exported variables, and external name refereces of the imported variables,

and to link an object file with

• ther ML object files independenly compiled, and

system libraries (libc, libm etc)

As far as I can see, separate compilation in this ordinary sense has never been done before for Standard ML or equivalent one. Indeed, separately compiling Standard ML declarations into an object file is inherently difficult.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 48 / 59

Outline of the Technical Developments Separate compilation

Issues in separately compiling Standard ML (I)

To compile a program separately from those it references, it must be closed both in its type and in its representation. Suppose we compile a source program that references a structure

S :> sig type foo val x : foo end

Since foo is unknown, we must specify its interface such as

structure S = struct type foo (= real * int) (* opaque type interface *) val x : foo end

Atsushi Ohori Development of SML# – making ML an ordinary practical language 49 / 59

Outline of the Technical Developments Separate compilation

Interface language

By this way, we design an interface language whose syntax is roughly:

<itopdec> ::= <idecl> | <ifundecl> <idecl> ::= val <valdesc> | type <itypbind> | eqtype <tyvarSeq> id (= <ty>) | datatype <idatbind> | datatype <id> = datatype <longid> | exception <exbind> | structure <id> = <istrexp> <itypbind> ::= <tyvarSeq> <id> = <ty> | <tyvarSeq> <id> ( = <ty> ) <idatbind> ::= <tyvarSeq> <id> = <conbinds> [and <idatbind>] | <tyvarSeq> <id> ( = <conbinds> [and<idatbind>]) <istrexp> ::= struct <idecl-seq> end <ifundec> ::= functor <id> ( <id> : <sigexp> ) = <istrexp>

Atsushi Ohori Development of SML# – making ML an ordinary practical language 50 / 59

Outline of the Technical Developments Separate compilation

Separately compiling a functor

Then how to deal with functor, which is by definition open in its type parameters.

functor F (X:sig type foo datatype bar = A of foo * int end) = struct datatype hoge = X of int * X.bar fun f (a, b) = X (a, X.A (b, a)) fun g (X (_, X.A c)) = c end

Since foo’s is a parameter, its representation varies.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 51 / 59

Outline of the Technical Developments Separate compilation

Lifting functor arguments

We have solved this problem based roughly on the following idea: lifting static (type) parameters as type abstractions, lambda lift dynamic parameters (variables) A functor becomes a polymorphic function and functor application become application with type instantiation. We can then use the idea of record compilation again to pass necessary compile attributes through type abstraction and type application. The example code is transformed to the code of the form:

val ’foo F = fn (id : ’foo -> ’foo) => let fun f (a, b) = ... fun g v = ... in (f, g) end

Atsushi Ohori Development of SML# – making ML an ordinary practical language 52 / 59

Outline of the Technical Developments Separate compilation

Other issues in separate compilation

There are a number of other issues that had to be worked out, including flattening nested structures (mostly mechanical), complete signature checking before type inference proper treatment of exceptions statically allocating constants and records in top-level declarations, generating initialization code, and generating system linkable object files. We have solved all the problems and have a separate compile system that compile any sequence of declarations.

The details will (hopefully) be published somewhere in near future, we hope.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 53 / 59

Outline of the Technical Developments Separate compilation

Separate compilation demo : lexgen

We have wrote interface file for each file in the basis library.

basis.smi include "basis/main/ArrayStructures.smi" include "basis/main/BinIO.smi" include "basis/main/Bool.smi" .... lexgen benchmark can be compiled by writing the interface file _require "basis.smi" structure Main = struct val doit : unit -> unit end

without consulting any of the actual source files. The resulting object file can be liked with those of the basis to form an executable file by the command line:

% smlsharp lexgen.smi

Atsushi Ohori Development of SML# – making ML an ordinary practical language 54 / 59

Outline of the Technical Developments Seamless database integration

Seamless database integration in SML#

SML# achieves seamless databases integration where SQL expressions are first-class citizens in the language that can be freely combined with all the other components of SML#, and they are evaluated by a database server. using the following results and new technical developments: database type system a la Machiavelli [Ohori, et.al. 1989, 1996], record polymorphism, C FFI, light-weight first-class overloading (through record compilation again), limited introduction of existential types, and a uniform approach to implement a DSL, bridging static and dynamic semantics.

The details at the talk tomorrow.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 55 / 59

Outline of the Technical Developments Efficient Non-moving GC

Efficient non-moving GC

It is absurd to assume that objects are moved at any unpredictable time. Indeed, there are many objects you can’t move....

Atsushi Ohori Development of SML# – making ML an ordinary practical language 56 / 59

Outline of the Technical Developments Efficient Non-moving GC

Efficient non-moving GC

It is absurd to assume that objects are moved at any unpredictable time. Indeed, there are many objects you can’t move.... My suggestion on a historical interest near the ICFP 2011 venue: Masakado-zuka, the resting place of Taira no Masakado’s head. Near the C-5 exit of Otemachi st. About 20

• min. walk from here.

Photography by Katsuhiro Ueno

Atsushi Ohori Development of SML# – making ML an ordinary practical language 56 / 59

Outline of the Technical Developments Efficient Non-moving GC

Efficient non-moving GC

It is absurd to assume that objects are moved at any unpredictable time. Indeed, there are many objects you can’t move.... My suggestion on a historical interest near the ICFP 2011 venue: Masakado-zuka, the resting place of Taira no Masakado’s head. Near the C-5 exit of Otemachi st. About 20

• min. walk from here.

Photography by Katsuhiro Ueno

Atsushi Ohori Development of SML# – making ML an ordinary practical language 56 / 59

Outline of the Technical Developments Efficient Non-moving GC

Efficient non-moving GC

It is absurd to assume that objects are moved at any unpredictable time. Indeed, there are many objects you can’t move.... My suggestion on a historical interest near the ICFP 2011 venue: Masakado-zuka, the resting place of Taira no Masakado’s head. Near the C-5 exit of Otemachi st. About 20

• min. walk from here.

Photography by Katsuhiro Ueno

Atsushi Ohori Development of SML# – making ML an ordinary practical language 56 / 59

Outline of the Technical Developments Efficient Non-moving GC

Efficient non-moving GC

If we are sure that objects once created will not be moved (as in an

• rdinary decent world), then interoperability with C and native thread

support become lot easier. We have successfully developed a non-moving GC with generational extension that is as efficient as the legendary Cheney’s generational copying collector.

The details at the talk this afternoon by Ueno san.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 57 / 59

Outline of the Technical Developments Efficient Non-moving GC

Conclusions

SML# is available from SML# 0.62 the interactive compiler SML# 0.90 the separate compiler (released on 2011-09-14) We hope to release SML# 1.00 in near future by integrating SML# 0.62 into SML# 0.90. Near Future Plan 64 bit support (easy) concurrent GC

• ptimization (type propagation)

Atsushi Ohori Development of SML# – making ML an ordinary practical language 58 / 59

Outline of the Technical Developments Efficient Non-moving GC

Thank You

Atsushi Ohori Development of SML# – making ML an ordinary practical language 59 / 59