1 Evaluating DSLs Richard Paige richard.paige@york.ac.uk 2 - - PowerPoint PPT Presentation

▶
1
SMART_READER_LITE
LIVE PREVIEW

1 Evaluating DSLs Richard Paige richard.paige@york.ac.uk 2 - - PowerPoint PPT Presentation

Sep 18, 2022 281 likes •454 views

1 Evaluating DSLs Richard Paige richard.paige@york.ac.uk 2 Overview How to evaluate a DSL? Observations: Different communities talk different criteria Fit for purpose, efficient, concise, acceptable by

slide-1
SLIDE 1

1

Evaluating DSLs

Richard Paige richard.paige@york.ac.uk

slide-2
SLIDE 2

2

Overview

  • How to evaluate a DSL?
  • Observations:

– Different communities talk different criteria

  • “Fit for purpose”, “efficient”, “concise”, “acceptable

by users” …

– Same label, different contents in the package.

  • “Readable”, “complete”

– Artistic or hedonic?

slide-3
SLIDE 3

3

Questions

  • We asked what you’d like to discuss here:

– Most of us are broadly interested in how to evaluate DSLs.

  • Some of the points raised:

– What terminology and criteria do we use? – What meanings are ascribed? – What might a successful empirical evaluation look like? – What might a successful formal/mathematical evaluation look like? – What might a successful user evaluation look like? – What might crappy evaluations look like?

slide-4
SLIDE 4

4

???

  • Some of us may have principles, ideas on

DSL evaluation …

– But none of us yet have answers. – Bits of related work, e.g., Amaral et al 2011, Hermans et al 2009.

  • Ripe for discussion!
slide-5
SLIDE 5

5

This Session

  • Pre-survey on DSL qualities.

– What’s going to happen next?

  • A short (10 min) talk about academic

evaluation of DSLs.

– No answers, just ideas.

  • Eric will talk about Cognitive dimensions.
slide-6
SLIDE 6

6

Pre-Survey

  • Some of you responded to an email survey on

qualities/evaluation of DSLs.

  • The questions:
  • 1. What are the three most important qualities a DSL

should possess?

  • 2. What (if any) is your favourite DSL? What are up to

three reasons why this is your favourite?

  • 3. What (if any) is your least favourite DSL? What are

up to three reasons why this is your least favourite?

  • NB, we are not looking for agreement or

consensus – we want your opinion.

– Still time to respond – please do, email to me.

slide-7
SLIDE 7

7

Pre-Survey

  • I’ll share the results later in the week (maybe

Weds morning or Thursday).

  • What happens now?
  • Posted names of DSLs in and around

meeting room (they will remain up).

– Using PostIts/chalk/pen, write down what is good

  • r bad about this DSL (if you have an opinion).
  • Use qualities already supplied
  • Define new qualities if you like!

– Take some time now, but will leave up the DSLs until Wednesday end of day.

slide-8
SLIDE 8

8

Follow-up

  • Invitation to on-line card sorting (possibly

Thursday evening, more likely next week).

  • Question we’re trying to answer: what do

DSL qualities (e.g., “readability”) mean to you?

– You’ll put qualities/terms into categories that make sense to you. – We will look for patterns!

  • Card sorting will take around 15 minutes.

– Results will be made available to all participants.

slide-9
SLIDE 9

9

Academic Evaluation of DSLs

aka ¡Marking/Grading ¡ aka ¡The ¡Worst ¡Thing ¡about ¡Academia ¡

slide-10
SLIDE 10

10

Academic Evaluation

  • We’ve taught DSL design and

implementation for >10 years.

– External DSLs, built using MDD technologies (EMF/Ecore/GMF).

  • How do we mark student submissions?

– Both undergraduate and MSc. – Again, no answers – this is what we do/have tried.

slide-11
SLIDE 11

11

External Criteria

  • “Fit for purpose”:

– Are there missing concepts? – Are there inexpressible “sentences” or patterns users need to express? – Does the DSL permit desired operations to be implemented against it? (efficiently, accurately..)

  • For example, does the DSL support the structures

needed to implement a code generator?

– Are the annotations/icons/tokens chosen for concrete syntax acceptable to domain experts and/or end-users?

slide-12
SLIDE 12

12

Internal Criteria

  • Does the DSL abstract syntax make use of

design patterns (eg, composite, façade)?

  • Are suitable naming conventions used for

language features?

  • Does the implementation load in an

appropriate tool?

  • Is each metaclass needed?

– A requirement, implementation (for metamodelling platform), or design?

slide-13
SLIDE 13

13

Internal Criteria

  • Is any duplication/redundancy in the

abstract syntax justifiable?

  • Is the DSL “habitable” [a la Richard

Gabriel].

– Has it been implemented in a way that facilitates change? – i.e., can humans “live” within the DSL and support structures.

  • Tricky: habitability will vary over time (as may

metrics of interest).

slide-14
SLIDE 14

14

Style vs Substance

  • Can we capture the difference between a

“working” DSL and a “beautiful working” DSL?

– Does it make appropriate use of (MDE) languages and tools, e.g., declarative vs imperative. – Appropriate use of metamodelling features (reference navigation, containment, generalisation). – Can others [students/teachers] understand design and rationale?

slide-15
SLIDE 15

15

Justification

  • Require justification and evidence.

– For decisions and design choices. – For process that was followed (eg, several increments). – For choices of patterns used. – For claims that requirements are supported.

  • Emphasis on justification in marking

scheme (e.g., 60%).

slide-16
SLIDE 16

16

Academic Evaluation

  • Treat the DSL like a software product.

– Justification of clearly described design decisions. – Fit for purpose. – “Beautiful” vs “just works”.