1 Evaluating DSLs Richard Paige richard.paige@york.ac.uk 2 - PowerPoint PPT Presentation

1 Evaluating DSLs Richard Paige richard.paige@york.ac.uk

2 Overview • How to evaluate a DSL? • Observations: – Different communities talk different criteria • “Fit for purpose”, “efficient”, “concise”, “acceptable by users” … – Same label, different contents in the package. • “Readable”, “complete” – Artistic or hedonic?

3 Questions • We asked what you’d like to discuss here: – Most of us are broadly interested in how to evaluate DSLs. • Some of the points raised: – What terminology and criteria do we use? – What meanings are ascribed? – What might a successful empirical evaluation look like? – What might a successful formal/mathematical evaluation look like? – What might a successful user evaluation look like? – What might crappy evaluations look like?

4 ??? • Some of us may have principles, ideas on DSL evaluation … – But none of us yet have answers. – Bits of related work, e.g., Amaral et al 2011, Hermans et al 2009. • Ripe for discussion!

5 This Session • Pre-survey on DSL qualities. – What’s going to happen next? • A short (10 min) talk about academic evaluation of DSLs. – No answers, just ideas. • Eric will talk about Cognitive dimensions.

6 Pre-Survey • Some of you responded to an email survey on qualities/evaluation of DSLs. • The questions: 1. What are the three most important qualities a DSL should possess? 2. What (if any) is your favourite DSL? What are up to three reasons why this is your favourite? 3. What (if any) is your least favourite DSL? What are up to three reasons why this is your least favourite? • NB, we are not looking for agreement or consensus – we want your opinion. – Still time to respond – please do, email to me.

7 Pre-Survey • I’ll share the results later in the week (maybe Weds morning or Thursday). • What happens now? • Posted names of DSLs in and around meeting room (they will remain up). – Using PostIts/chalk/pen, write down what is good or bad about this DSL (if you have an opinion). • Use qualities already supplied • Define new qualities if you like! – Take some time now, but will leave up the DSLs until Wednesday end of day.

8 Follow-up • Invitation to on-line card sorting (possibly Thursday evening, more likely next week). • Question we’re trying to answer: what do DSL qualities (e.g., “readability”) mean to you? – You’ll put qualities/terms into categories that make sense to you . – We will look for patterns! • Card sorting will take around 15 minutes. – Results will be made available to all participants.

9 Academic Evaluation of DSLs aka ¡Marking/Grading ¡ aka ¡The ¡Worst ¡Thing ¡about ¡Academia ¡

10 Academic Evaluation • We’ve taught DSL design and implementation for >10 years. – External DSLs, built using MDD technologies (EMF/Ecore/GMF). • How do we mark student submissions? – Both undergraduate and MSc. – Again, no answers – this is what we do/have tried.

11 External Criteria • “Fit for purpose”: – Are there missing concepts? – Are there inexpressible “sentences” or patterns users need to express? – Does the DSL permit desired operations to be implemented against it? (efficiently, accurately..) • For example, does the DSL support the structures needed to implement a code generator? – Are the annotations/icons/tokens chosen for concrete syntax acceptable to domain experts and/or end-users?

12 Internal Criteria • Does the DSL abstract syntax make use of design patterns (eg, composite, façade)? • Are suitable naming conventions used for language features? • Does the implementation load in an appropriate tool? • Is each metaclass needed? – A requirement, implementation (for metamodelling platform), or design?

13 Internal Criteria • Is any duplication/redundancy in the abstract syntax justifiable? • Is the DSL “habitable” [a la Richard Gabriel]. – Has it been implemented in a way that facilitates change? – i.e., can humans “live” within the DSL and support structures. • Tricky: habitability will vary over time (as may metrics of interest).

14 Style vs Substance • Can we capture the difference between a “working” DSL and a “beautiful working” DSL? – Does it make appropriate use of (MDE) languages and tools, e.g., declarative vs imperative. – Appropriate use of metamodelling features (reference navigation, containment, generalisation). – Can others [students/teachers] understand design and rationale?

15 Justification • Require justification and evidence. – For decisions and design choices. – For process that was followed (eg, several increments). – For choices of patterns used. – For claims that requirements are supported. • Emphasis on justification in marking scheme (e.g., 60%).

16 Academic Evaluation • Treat the DSL like a software product. – Justification of clearly described design decisions. – Fit for purpose. – “Beautiful” vs “just works”.