INTEGRATING IDE INTEGRATING IDEs WITH DOTTY WITH DOTTY Guillaume - - PowerPoint PPT Presentation

â–¶
integrating ide integrating ides with dotty with dotty
SMART_READER_LITE
LIVE PREVIEW

INTEGRATING IDE INTEGRATING IDEs WITH DOTTY WITH DOTTY Guillaume - - PowerPoint PPT Presentation

Jun 27, 2023 22 likes •553 views

INTEGRATING IDE INTEGRATING IDEs WITH DOTTY WITH DOTTY Guillaume Martres - EPFL 1 WHAT IS DOTTY? WHAT IS DOTTY? Research compiler that will become Scala 3 Type system internals redesigned, inspired by DOT, but externally very similar

slide-1
SLIDE 1

INTEGRATING IDE INTEGRATING IDEs WITH DOTTY WITH DOTTY

  • EPFL

Guillaume Martres

1

slide-2
SLIDE 2

WHAT IS DOTTY? WHAT IS DOTTY?

Research compiler that will become Scala 3 Type system internals redesigned, inspired by DOT, but externally very similar More info: Recent blog posts on dotty.epfl.ch scala-lang.org

2

slide-3
SLIDE 3

A CHANCE TO REDESIGN A CHANCE TO REDESIGN COMPONENTS COMPONENTS

Improved incremental compilation (avoid undercompilation) Better pattern matching checks ( ) algorithm now reused in Swi!

3

slide-4
SLIDE 4

TOOLING TOOLING

A good developer experience requires good tools: A REPL (with syntax highlighting!) Dottydoc (used to generate ) IDE support dotty.epfl.ch/docs

4

slide-5
SLIDE 5

STATE OF THE ART STATE OF THE ART

Based on the Scala-IDE ENSIME Reimplementation of the Scala typechecker Scala plugin for IntelliJ IDEA Scala Presentation Compiler

5

slide-6
SLIDE 6

STATE OF THE ART STATE OF THE ART

Based on the (3 KLOC) Scala-IDE (66 KLOC) ENSIME (server: 15 KLOC, emacs client: 10 KLOC) Reimplementation of the Scala typechecker Scala plugin for IntelliJ IDEA (230 KLOC) Scala Presentation Compiler

6

slide-7
SLIDE 7

DESIGN PRINCIPLES DESIGN PRINCIPLES

  • 1. Code reuse
  • 2. Editor-agnosticity
  • 3. Easy to use (and to

install!)

7

slide-8
SLIDE 8

DESIGN PRINCIPLES DESIGN PRINCIPLES

  • 1. Code reuse

2. 3. Editor-agnosticity Easy to use (and to install!)

8

slide-9
SLIDE 9

QUERYING THE COMPILER QUERYING THE COMPILER

Each phase progressively simplify trees until they can be emitted as JVM bytecode

9

slide-10
SLIDE 10

QUERYING THE COMPILER QUERYING THE COMPILER

Each phase progressively simplify trees until they can be emitted as JVM bytecode

10

slide-11
SLIDE 11

SOURCE CODE SOURCE CODE

Cursor position = 🚪 Query: jump to definition

final val elem = 1 val foo = elem🚪 + 1

11

slide-12
SLIDE 12

TREE AFTER TREE AFTER Typer Typer

Every tree node has a type and a position Query can be answered

final val elem: 1 = 1 val foo: Int = elem + 1

12

slide-13
SLIDE 13

TREE AFTER TREE AFTER FirstTransform FirstTransform

Information lost by constant folding Impossible to answer query

final val elem: 1 = 1 val foo: Int = 2

13

slide-14
SLIDE 14

QUERYING THE COMPILER QUERYING THE COMPILER

Store trees before FirstTransform Respond to IDE queries by traversing trees What about code that has already been compiled?

14

slide-15
SLIDE 15

PICKLING PICKLING

In Scala 2: store methods signatures (for separate compilation) In Dotty: store full trees

15

slide-16
SLIDE 16

TASTY: TYPED AST TASTY: TYPED AST SERIALIZATION FORMAT SERIALIZATION FORMAT

Original motivation: solve the Always use JVM bytecode: breaks when compiler encoding changes Always recompile source code: breaks when the typechecker changes Can also be used to provide interactive features: deserialize and query trees binary compatibility problem

16

slide-17
SLIDE 17

INTERACTIVE APIS INTERACTIVE APIS

Convenience methods for tree traversals, compiler lifecycle management Used both in the IDE and the REPL (e.g., for completions) In the future: interruption handling, partial typechecking, ... Less then 1 KLOC

17

slide-18
SLIDE 18

DESIGN PRINCIPLES DESIGN PRINCIPLES

1.

  • 2. Editor-agnosticity

3. Code reuse Easy to use (and to install!)

18

slide-19
SLIDE 19

THE IDE PORTABILITY PROBLEM THE IDE PORTABILITY PROBLEM

Getting m IDEs to support n programming languages requires n*m IDE plugins.

19

slide-20
SLIDE 20

THE LANGUAGE SERVER THE LANGUAGE SERVER PROTOCOL PROTOCOL

20

slide-21
SLIDE 21

BASICS OF THE LSP BASICS OF THE LSP

First implemented in Visual Studio Code JSON-RPC IDE notifies the language server about user actions LS maintains internal representation of code LS notify IDE about warnings/errors IDE can send requests usually triggered by user actions Asynchronous, cancelable

21

slide-22
SLIDE 22

IMPLEMENTING THE DOTTY IMPLEMENTING THE DOTTY LANGUAGE SERVER LANGUAGE SERVER

Low-level message handling done by Relies on interactive APIs 0.5 KLOC Eclipse LSP4J

22

slide-23
SLIDE 23
  • verride def definition(params: TextDocumentPositionParams) =

}

23

slide-24
SLIDE 24
  • verride def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken => }

24

slide-25
SLIDE 25
  • verride def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken => val uri = new URI(params.getTextDocument.getUri) }

25

slide-26
SLIDE 26
  • verride def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken => val uri = new URI(params.getTextDocument.getUri) val driver = driverFor(uri) }

26

slide-27
SLIDE 27
  • verride def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken => val uri = new URI(params.getTextDocument.getUri) val driver = driverFor(uri) implicit val ctx = driver.currentCtx }

27

slide-28
SLIDE 28
  • verride def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken => val uri = new URI(params.getTextDocument.getUri) val driver = driverFor(uri) implicit val ctx = driver.currentCtx val pos = sourcePosition(driver, uri, params.getPosition) }

28

slide-29
SLIDE 29
  • verride def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken => val uri = new URI(params.getTextDocument.getUri) val driver = driverFor(uri) implicit val ctx = driver.currentCtx val pos = sourcePosition(driver, uri, params.getPosition) val uriTrees = driver.openedTrees(uri) }

29

slide-30
SLIDE 30
  • verride def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken => val uri = new URI(params.getTextDocument.getUri) val driver = driverFor(uri) implicit val ctx = driver.currentCtx val pos = sourcePosition(driver, uri, params.getPosition) val uriTrees = driver.openedTrees(uri) val sym = Interactive.enclosingSourceSymbol(uriTrees, pos) }

30

slide-31
SLIDE 31
  • verride def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken => val uri = new URI(params.getTextDocument.getUri) val driver = driverFor(uri) implicit val ctx = driver.currentCtx val pos = sourcePosition(driver, uri, params.getPosition) val uriTrees = driver.openedTrees(uri) val sym = Interactive.enclosingSourceSymbol(uriTrees, pos) val classTree = SourceTree.fromSymbol(sym.topLevelClass.asClass).toList }

31

slide-32
SLIDE 32
  • verride def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken => val uri = new URI(params.getTextDocument.getUri) val driver = driverFor(uri) implicit val ctx = driver.currentCtx val pos = sourcePosition(driver, uri, params.getPosition) val uriTrees = driver.openedTrees(uri) val sym = Interactive.enclosingSourceSymbol(uriTrees, pos) val classTree = SourceTree.fromSymbol(sym.topLevelClass.asClass).toList val defTree = Interactive.definition(classTree, sym) }

32

slide-33
SLIDE 33
  • verride def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken => val uri = new URI(params.getTextDocument.getUri) val driver = driverFor(uri) implicit val ctx = driver.currentCtx val pos = sourcePosition(driver, uri, params.getPosition) val uriTrees = driver.openedTrees(uri) val sym = Interactive.enclosingSourceSymbol(uriTrees, pos) val classTree = SourceTree.fromSymbol(sym.topLevelClass.asClass).toList val defTree = Interactive.definition(classTree, sym) defTree.map(d => location(d.namePos)).asJava }

33

slide-34
SLIDE 34

DESIGN PRINCIPLES DESIGN PRINCIPLES

1. 2.

  • 3. Easy to use (and to

install!) Code reuse Editor-agnosticity

34

slide-35
SLIDE 35

SBT INTEGRATION SBT INTEGRATION

Analyze the build to find Dotty projects Compile these projects Generate configuration files Install the Dotty VSCode extension Launch VSCode

35

slide-36
SLIDE 36

CONFIGURATION FILES CONFIGURATION FILES

.dotty-ide-artifact, used by the IDE extension to launch the Dotty Language Server:

ch.epfl.lamp:dotty-language-server_0.8:0.8.0-RC1

36

slide-37
SLIDE 37

CONFIGURATION FILES CONFIGURATION FILES

.dotty-ide.json, used by the DLS to launch compiler instances:

[ { "id" : "root/compile", "compilerVersion" : "0.8.0-RC1", "compilerArguments" : [ ], "sourceDirectories" : [ "src/main/scala" ], "dependencyClasspath" : [ ... ], "classDirectory" : "target/scala-0.8/classes" }, { "id" : "root/test", ... }, ... ]

37

slide-38
SLIDE 38

BUILD SERVER PROTOCOL BUILD SERVER PROTOCOL

Instead of making plugins for build tools to extract information, ask them! We also need a discovery protocol: "How do I start a build server for this project?"

38

slide-39
SLIDE 39

DESIGN PRINCIPLES, REVISITED DESIGN PRINCIPLES, REVISITED

  • 1. Code reuse

Compiler APIs for interactive usage

  • 2. Editor-agnosticity

Implemented the LSP

  • 3. Easy to use (and to install!)

One command (but we can do better!)

39

slide-40
SLIDE 40

GOING FURTHER: DEBUGGER GOING FURTHER: DEBUGGER SUPPORT SUPPORT

Based on the Most features "just work" (Not actually merged in Dotty yet) Challenge: expression evaluation Java Debug Server

40

slide-41
SLIDE 41

EXPRESSION EVALUATION EXPRESSION EVALUATION

class Hello { def foo(implicit y: Context): String = { /*...*/ } def bar(implicit y: Context) = { /*...*/ } }

41

slide-42
SLIDE 42

EXPRESSION EVALUATION EXPRESSION EVALUATION

class Hello { def foo(implicit y: Context): String = { /*...*/ } def bar(implicit y: Context) = { 🚪 /*...*/ } }

42

slide-43
SLIDE 43

EXPRESSION EVALUATION EXPRESSION EVALUATION

class Hello { def foo(implicit y: Context): String = { /*...*/ } def bar(implicit y: Context) = { 🚪 foo /*...*/ } }

43

slide-44
SLIDE 44

RUN THE COMPILER PIPELINE RUN THE COMPILER PIPELINE

class Hello { def foo(y: Context): String = { /*...*/ } def bar(y: Context) = { 🚪 this.foo(y) /*...*/ } }

44

slide-45
SLIDE 45

EXTRACT TO A STATIC METHOD EXTRACT TO A STATIC METHOD

  • bject Global {

def liftedExpr($this: Hello, y: Context) = $this.foo(y) }

45

slide-46
SLIDE 46

EXTRACT TO A STATIC METHOD EXTRACT TO A STATIC METHOD

  • bject Global {

def liftedExpr($this: Hello, y: Context) = $this.foo(y) def exec(self: Object, localVariables: Map[String, Object]) = liftedExpr( self.asInstanceOf[Hello], localVariables("y").asInstanceOf[Context] ) }

46

slide-47
SLIDE 47

ON THE DEBUGGING VM ON THE DEBUGGING VM

Compile Global to a classfile Load it in the debugged VM Call Global.exec with the right arguments

47

slide-48
SLIDE 48

ON THE DEBUGGED VM ON THE DEBUGGED VM

In the standard library

package dotty.runtime

  • bject DebugEval {

def eval(classpath: String, self: Object, localVariables: Map[String, Object]): Any = { val cl = new URLClassLoader(Array(new URL("file://" + classpath) val cls = cl.loadClass("Global") val instance = cls.newInstance val exec = cls.getMethod("exec") exec.invoke(instance, self, names, args) } }

48

slide-49
SLIDE 49

ON THE DEBUGGING VM ON THE DEBUGGING VM

We want to remotely execute:

val classpath = <classpath for the compiled Global class> val self = <this in the stackframe> val localVariables = <map of local variables in the stackframe> dotty.runtime.DebugEval(classpath, self, localVariables)

49

slide-50
SLIDE 50

ON THE DEBUGGING VM ON THE DEBUGGING VM

We use the Java Debugging Interface APIs:

val vars = stackFrame.visibleVariables val mapCls = vm.classesByName("java.util.Map") .get(0).asInstanceOf[ClassType] val localVariablesRef = mapCls.newInstance() // Skipped: store `vars` into `localVariablesRef` val debugCls = vm.classesByName("dotty.runtime.DebugEval") .get(0).asInstanceOf[ClassType] val eval = debugCls.methodsByName("eval").get(0) debugCls.invokeMethod( thread, eval, List(vm.mirrorOf(classpath), stackFrame.thisObject, localVariablesRef) )

50

slide-51
SLIDE 51

FUTURE WORK FUTURE WORK

Optimizations More features Documentation on hover Better build tool integration (Build Server Protocol!)

51

slide-52
SLIDE 52

CONCLUSION CONCLUSION

Design your compiler with interactivity in mind Design your build tool with interactivity in mind Interactivity should go beyond what IDEs and REPLs currently offer Type Driven Development with Idris

52

slide-53
SLIDE 53

QUESTIONS ? QUESTIONS ?

More info: Come chat with us: Contributors welcome! dotty.epfl.ch gitter.im/lampepfl/dotty

53