Archive for the 'Thesis' Category

How much of total software cost goes to maintenance?

September 8, 2009

I thought I had a simple question: How much of total software cost goes to maintenance? I want a number that is recent, backed by evidence, and generally citation-worthy.

It turns out it’s hard to find. Software engineering course notes online tend not to cite their sources. Recent estimates tend to come from magazine articles citing Gartner or Forrester. A 2000 IEEE-published article was cited as “90%,” but digging into the paper I find:

According to an informal industry poll,85 to 90 percent of IS budgets goes to legacy system operation and maintenance.

If you know where I should look, please get in touch.

Advertisements

Case Study Databases vs. Ethics and Participation

August 5, 2009

Robert Yin advocates the creation of a case study database alongside the publication of  case study. In my interpretation of this idea, the case study database includes the raw data collected, such as transcripts, interview recordings, and documents collected from archives.  The creation of an “analysis-free” data repository allows others to reanalyze the data to reach different conclusions or for new purposes. This potential for reanalysis in turn strengthens the case study.

For my study, that would mean publishing my interview recordings, my transcriptions of those recordings, and the statistics about software repositories that I collect.

Greg points out two issues with this that have caused me to reverse my plan.

  1. This open approach is rare enough that it might have trouble getting through the ethics board. The withdrawal protocols become awkward, for instance.
  2. This might discourage participation. People might not want to be interviewed if the interview recording will be made available.

I think I will try with a new policy:

  1. Raw data (interview recordings) will be kept private and secure immediately after recording. They will be destroyed after publication of an analysis.
  2. Objective data such as interview transcripts will be kept private and secure. They will be destroyed after publication of an analysis.
  3. A summary of the interviews and other derivative data will be publicly available. If a subject requests it, this summary will be anonymized and the subject will be asked to approve the anonymized version prior to publication. Subject withdrawal will result in destruction of the summary if it has not yet been published. If it has already been published then withdrawal is not possible.

Theory product

July 9, 2009

For each item on the left what does the theory on the right predict about DSLs as a consequence?

Reformulated, Well-Formed Research Questions

July 3, 2009

According to the Case Study Research book, my research questions should be “well-formed.” This means they are of the form “Who, What, When, Where, Why, How, How Much.”

I have translated my research questions into this form below.

It is clear that many of my questions are not best answered with a case study. It is also clear that some of my questions can only be answered in a case study.

I think someone could spend a whole career finding quantitative answers to quantitative questions below, but I’m not sure it would be worth it. These questions ask only for credible answers, not conclusive answers.

  • What are the differences in maintenance between DSL programs and GPL programs?
    • How much maintenance effort is spent on DSL programs?
      • How much effort is spent on debugging DSL programs?
        • Why does the amount of debugging effort for DSL programs differ from the amount of debugging effort for similar-sized GPL programs?
        • What are the differences in the nature of bugs between DSL programs and GPL programs?
          • How do programmers form mental models of DSL program execution?
          • What is the accuracy of these mental models?
            • How much difficulty do programmers have forming cognitive models of alternative execution models? Why?
            • How are programmers trained in specific DSLs?
          • What are the differences in difficulty caused by tool differences?
          • How much do GPL tools help with DSL maintenance?
          • What are the differences in difficulty caused by differences in size between DSL programs and GPL programs?
            • How do we compare the size of a DSL program to the size of a GPL program?
            • Why should or shouldn’t we compare by functionality?
            • Why should or shouldn’t we compare by effort invested in creation?
            • Why should or shouldn’t we compare by physical size, i.e. bytes/tokens/lines?
        • How do programmers debug DSL programs?
        • How much do programmers follow a known model of GPL debugging, e.g. Hale and Haworth 1991?
        • How much do programmers use known GPL debugging tactics and strategies on DSL programs?
    • How much effort is spent on change of DSL programs to meet new requirements?

Outline of a Literature Review

July 3, 2009
  1. DSLs are important
    1. Definitive ANTLR reference positioned as book teaching how to make DSLs
    2. Sprinkle et al. Guest Editor introduction in latest IEEE software on DSLs
    3. S. Dmitriev “Language Oriented Programming: The Next Programming Paradigm”
  2. DSLs
    1. What is a DSL
      1. Definitions (Variety of sources)
      2. Bentley 1986 Little Languages
    2. How DSLs are implemented
      1. Supporting the DSL Spectrum (review of DSL approaches)
      2. When and how to develop domain-specific languages
      3. Notable design patterns for DSLs
    3. DSL benefits
      1. Supporting the DSL Spectrum (claimed 50:1 improvement in program length)
    4. Why DSLs work
      1. SWYN: Notation has a large effect on performance. Mental models of evaluators.
      2. When and how to develop domain-specific languages
      3. Prechelt 2000/2003: Scripting langs. vs. stodgier langs empirical comparison
    5. Misc. Surveys of other DSL literature
      1. Domain-specific languages in software development
      2. Domain-specific languages: An annotated bibliography
      3. Landin 1966: The next 700 programming languages
      4. Beyond LEX and YACC: Introduction to a number of DSLs to simplify compiler construction
      5. Fowler 2005 on language workbenches
  3. Debugging
    1. Maintenance is expensive and important
      1. Measuring and managing software maintenance
    2. There is a variety of data on debugging strategies and tactics and tools to support these
      1. Weiser 1982: Programmers use slices when debugging (just a tactic)
      2. Gugerty and Olson 1986 Debugging by Skilled and Novice Programmers (not much theory)
      3. Eisenstadt 1993: Tales of debugging from the front lines
      4. Murphy 2008 Debugging–a qualitative analysis of novices’ strategies
      5. Ko 2005, Ko 2006: Detailed study of corrective and perfective maintenance tasks, how it informs Whyline/IDE design
      6. Ko 2006: Debugging by asking q. about program output (Introduction of the Whyline)
      7. Liu 2006: Statistical Debugging: A hypothesis testing based approach
      8. Ruthruff 2005: Empirical study of fault localization for end-user programmers
      9. Ducasse and Emde 1988 A review of automated debugging systems (some debugging strategies and types of knowledge)
    3. There are a variety of models of bugs and debugging
      1. Vessey 1989 Toward a theory of computer program bugs (debugging is about program comprehension)
      2. Hale 1991 Towards a model of programmers’ cognitive processes during software maintenance, Hale 1999 evaluation of the model (Structural Learning Theory)
      3. Eisenstadt 1993: Tales of debugging from the front lines
      4. Von Mayrhauser/Vans 1997/1999: Program understanding behavior during maintenance (Integrated Comprehension Model)
      5. Xu 2004: Cognitive processes during program debugging (Bloom’s Taxonomy in the Cognitive Domain)
      6. Ramanujan 2000: An experimental investigation of the impact of individual, program, and organizational characteristics on software maintenance effort (Human Information Processing)
    4. Misc survey
      1. Debugging: A review of the literature (McCauley 2008)
  4. DSLs + Debugging: Nearly green-field
    1. There is a need for DSL debuggers
      1. Weaving a debugging aspect into DSL etc.
      2. KHEPERA
      3. Editorial Review of above: Cost of maintenance + proliferation of DSL usage = DSL maintenance
    2. Survey of literature looking at DSL maintenance
      1. Little Languages: Little maintenance?
      2. DSL maintenance vs. UML maintenance in IEEE software
      3. A software engineering experiment in software component generation: Experiment on DSL vs. GPL tasks
    3. Editorial Review: Most published research on debugging in general does not apply to DSLs in general. Not much literature on DSL maintenance
    4. Need for theory-based approach
      1. Hannay 2007 A Systematic Review of Theory Use in Software Engineering Experiments
      2. Ko 2005: Framework and methodology for studying the causes of errors in programming systems (theory of bug introduction)

This justifies and leads into my thesis, which, in the current plan:

  • Extends Ko’s 2005 theory to make predictions about DSL bugs (by incorporating knowledge of what DSLs are, when they are used and why they work)
  • Applies the various theories of debugging to DSLs or shows why they are difficult to apply
  • Claims that there is virtually no existing research on costs of DSL maintenance
  • Claims that there is very little understanding of DSL maintenance issues
  • Predicts that DSL maintenance is an important consideration
  • Examines the above predictions via interviews of DSL-using practitioners in Toronto.

Theories of DSL Debugging

June 23, 2009

The ideal case study builds and validates a theory around its evidence. For my thesis topic, that means I would want to find a strong predictive and explanatory theory of debugging, generalize it to DSLs, and evaluate whether it explains or does not explain the data I find in practice.

One thing I am noticing is that many of the “theories of debugging” I have found are descriptive and somewhat explanatory but not really predictive— it isn’t clear how to generalize them to make predictions about DSL maintenance. This is part of the general fact that ESE theory use is underdeveloped. I’ve been thinking about the relevance of this challenge and how to work around it, but no good ideas yet.

More Notes on the Case Study Research Method

June 21, 2009

After I posted my notes on Robert Yin’s “Case Study Research”, Greg replied with no fewer than ten comments. Once I got over my initial !@#$ reaction, I was glad because the questions made me think more carefully about the material.

What (in practice) is the difference between Exploratory, Descriptive or Explanatory purposes?

Exploratory: The goal is to develop propositions for further study.

Descriptive: The goal is to describe something unknown.

Explanatory: The goal is to make causal conclusions about why a decision or event happened as it did. There is an intention that the results generalize to other situations.

Thus the difference between the purposes is a difference in goals and research questions.

Do you have to have a theory before you can start a case study? Or is the purpose of an exploratory case study to (help) develop a theory?

In descriptive and explanatory studies, one starts with propositions. In an exploratory study, we may not start with propositions. Instead, we state the purpose and the success/exit criteria of the exploration.

How do you tell if you have the characteristics of a good case study investigator & are applying them?  Most people think they’re good listeners, but are wrong.

There is no test. An investigator must be honest with himself/herself and work to remedy any weaknesses.

You quote “Nor, in many instances, do case study reports end up in journals” 😦  Does the book explain why not?  And is that actually the case in empirical software engineering?

Why aren’t case study reports always published in journals? What happens to case studies in Empirical Software Engineering?

There is no commonly accepted outline format for a case study reports. Many case studies are published as books, or they are commissioned reports. They are much larger than typical experiments.

Any suggestions on the design of a case study database?  Templates you could clone?

A case study database is not like a computer database. Instead, it is simply the name for the idea of separating the raw data from the presentation of the data in the final report. Within reason, a reader ought to be able to go back and read the documents, interviews and other information that informed the report.

Do you need to have a theory to test? What do you do in the Exploratory case?

Exploratory case studies replace theory propositions with goals and criteria for success.

In your summary, what do you mean by a “whether” question?

This is me getting into trouble with badly-formed questions. I don’t trust case studies to tell me “whether” an intervention had a desired effect (well-formed: “What is the effect of the intervention?”).

In your summary, you say, “Case studies seem to be used in policy  to evaluate programs and decisions. I don’t think this is valid.”  Yin and others clearly think it *is* valid — why have they reached a different conclusion from you?

I think Yin would agree that “What”, “How much” and other questions are best addressed with other research methods. I’m not sure what Yin would say on the use of case studies for evaluation. I think case studies can provide great insight into why a program or decision works, but I would never trust a case study investigator who tells me a program is working because he or she interviewed some beneficiaries.

On matters of causality and bias, it is Yin’s opinion that, with sufficient rigour, case studies can be as reliable as other methods such as experiments, especially since experiments are also subject to bias. I just don’t agree. It would take a rather large experiment to determine whether decisions made informed by case studies were better than decisions made informed by quantitative methods.

In your summary, you say, “Several of my questions seem better addressed by surveys.”  Doesn’t a survey pre-suppose a strong theory?

Not necessarily. For example, surveys can ask “How Much” questions. I want to know “how much” maintenance is done on DSL programs. I can get at this by asking some companies about it in interviews, but an answer from a survey of many companies would be stronger evidence in my opinion.

Notes on “Case Study Research Design and Methods”

June 18, 2009

Chapter 1: Applicability of Case Studies and How to Do Them

Defn:

A case study is an empirical enquiry that investigates a contemporary phenomenon in depth and within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident. The case study inquiry copes with the technically distinctive situation in which there will be many more variables of interest than data points, and as one result relies on multiple sources of evidence, with data needing to converge in a triangulating fashion, and as another result benefits from the prior development of theoretical propositions to guide data collection and analysis.

Performing a case study entails:

  • Literature Review
  • Posing research questions
  • formal and explicit procedures
  • Protecting against threats to validity
  • Maintaining “chain of evidence”
  • Investigating and testing rival explanations
In advance of doing a case study:
  • Define the “case” being studied
  • Determine relevant data to be collected
  • What to do with data once collected

Decide: Exploratory, Descriptive or Explanatory research purposes?

How to choose a research method given a research question type:

  • How, Why: Experiment iff controllable events, else history or case study
  • Who, What, Where, How Many, How Much: Survey, Archival Analysis

Case study applicability: “How” or “Why” questions, in-depth explanation of social phenomena, contemporary events, little or no control over events.

Well-formed research questions = Who, What, When, Where, Why, How
Choose research method based on RQ’s, then reformulate RQ’s to match the research method.

Chapter 2: Research Design

Components of research design:

  • Study’s questions
  • Propositions
  • Units of analysis
  • Logic linking data to propositions
  • Criteria for interpreting the findings

Research Design Quality (items covered in detail below)

  • Construct Validity
    • Correct operational measure for concept being studied
    • Use multiple sources of evidence
    • Establish chain of evidence
    • Key informants review case study report drafts
  • Internal Validity
    • Causal relationships established
    • Pattern matching
    • Explanation building
    • Address rival explanations
    • Logic models
  • External Validity
    • Generalizing to what domain
    • Theory and replication logic
      • Literal replication across cases = same procedure -> same results
      • Theoretical replication across cases = different procedure -> predictably different results
  • Reliability
    • Reproducible
    • Case study protocol
    • Case study database

Keep in mind: Multiple cases != sampling. Different logical basis for generalization.

Chapter 3

A good case study investigator:

  • Ask good questions
  • Good listener
  • Adaptive and flexible
  • Firm grasp of issues being studied
  • Unbiased by preconceived notions

Human Subjects Protection:

  • Informed consent: Informed as to nature of study, formal record of volunteerism
  • Protection from harm, avoidance of deception
  • Protection of privacy and confidentiality
  • Protection of especially vulnerable groups

As pretraining, review:

  • Why the study is being done
  • What evidence is being sought
  • What variations can be anticipated
  • What would constitute supportive or contrary evidence for any proposition

Case study protocol: Per case instructions for the case study investigator.

  • Project goals, case study issues, readings
  • Field procedures:
    • Access & contact info for case site, procedural reminders
    • Sufficient resources when in the field
    • Procedure for calling for assistance
    • Clear schedule of data collection activities
    • Provisions for changes in mood and motivation of investigator
  • Case study questions to keep investigator on track, linked to potential sources of information
  • Levels of question targets:
    • Specific interviewees
    • Individual case <- focus of case study protocol
    • Pattern of findings across multiple cases
    • Entire study
    • Conclusions
  • Guide for case study report: outline, data formats, other documentation, bibliographical information

“Nor, in many instances, do case study reports end up in journals” 😦

Pilot case study

Refine data collection plans and procedures. Not a pretest. Formative. Preferably before final IRB approval. Usually a close, tolerant contact of an investigator.

Consider 20-30 candidate case studies, filter down to a few using well-defined criteria.

Chapter 4

Six common sources of evidence:

  1. Documents
  2. Archival records
  3. Interviews
  4. Direct observation
  5. Participant-observation
  6. Physical artifacts

Interviews

  • In-depth interview: Facts and opinions of key players
  • Focused interview: Short, mostly follow set of questions from case-study protocol. Corroborate evidence.
  • Formal survey: Quantitative data

Record if desired, but: Ask permission, ensure comfort of interviewee. Plan to transcribe recordings. Ensure facility with recorder.

Three principles of data collection

  1. Multiple sources of evidence
    1. Data triangulation
    2. Investigator triangulation
    3. Theory triangulation
    4. Methodological triangulation
  2. Create a case study database: Separation of collected data from reports.
    1. Case study notes
    2. Case study documents
    3. Tabular materials
    4. Narratives
  3. Maintain a chain of evidence
    1. Link report to evidence, evidence to protocol, protocol to study question.

Chapter 5: Analysis

Four General Analytic Strategies

  • Rely on theoretical propositions
  • Develop a case description
  • Use both qualitative and quantitative data
  • Examine rival explanations

Rival explanations:

  • Craft Rivals
    • Null Hypothesis: Observation result of chance
    • Threats to validity
    • Investigator bias
  • Real-Life Rivals
    • Direct rival: Other intervention had the effect
    • Commingled rival: Other interventions contributed to the result
    • Implementation rival: The implementation process, not the intervention, had the effect
    • Rival theory: A theory different from the original explains the result
    • Super rival: A force larger than (and including) the intervention accounts for the result
    • Societal rival: Social trends account for the results

Five analytic techniques

  • Pattern matching: Predicted outcomes (for all theories including rivals) vs. actual results
  • Explanation building: Iterative pattern matching
  • Time-series analysis: Comparing critical intervention timing with critical event timing. Building a chronology of events.
  • Logic Models: Causally-connected pattern matching.
  • Cross-case synthesis: Generalize commonalities from multiple cases

Chapter 6: Composing the report

Composing the case study report:

Target to the audience. Consider multiple reports if different audiences.

e.g. if thesis dissertation committee is audience, cite committee member’s work.

Four formats:

  • Single case report (narrative)
  • Multiple cases presented singly as narratives + chapter on cross-case analysis
  • Question and Answer format
  • Entire report is cross-case analysis. Chapters dedicated to separate cross-case issues.

Six compositional structures:

  • Linear-analytic (typical structure.) Problem, design, conclusions, implications.
  • Comparative. Repetitions of same case from different perspectives
  • Chronological. Written in order of real-world events.
  • Theory-building. Order dependent on theory.
  • Suspense. Inversion of linear-analytic. Shocking outcome presented first, followed by details of how it came to happen.
  • Unsequenced. No importance to sequence of chapters.

Start writing early. Literature review, case design -> bibliography, methodology.

After data collection, write descriptive data section (even before analysis).

Identify case study subjects unless ethical concerns preclude it. Anonymizing is lots of work.

Have informants review drafts.

Exemplary case studies:

  • Individual cases are of general public interest
  • Underlying issues are important
  • Case study is complete. Boundaries between phenomenon and context are clear. Investigator pursued all evidence, especially that would support rival explanations. Case study not cut short by resource or other constraints.
  • Consideration of alternative perspectives
  • Display sufficient evidence so that reader can come to own conclusions just from the evidence.
  • Composed in an engaging manner.

Overall Impressions

  • Yin spends a lot of effort justifying the validity of case studies, but I think it really comes down to the research questions. Case studies can help answer “how” or “why” questions but I don’t trust them for “whether” or other types of questions. Case studies seem to be used in policy  to evaluate programs and decisions. I don’t think this is valid. I think more solid empirical methods are needed to evaluate whether a program is achieving its goals. Case studies should be used to understand how and why (or why not).
  • Case studies are a lot of work. Yin says this a few times, actually, claiming that case studies are the most difficult of the research methods
  • I’m not sure how well my intended topic matches the case study method. Several of my questions seem better addressed by surveys.
  • The book was remarkably dry. Unnecessarily so in my opinion. It’s an instructional tutorial, and I don’t see why it needs to be written in a formal academic style.
  • I want to talk to someone who is good at case studies to find out what transferable skills I can expect to learn by pouring myself into this. The work/reward ratio for this thing seems pretty bad to me. I need to find some good extrinsic motivation.

Opportunities for Case Studies

Reading the book made me think of several possible interesting case study topics, one not software engineering.

  • The decision by the Athletic Centre to institute women’s only hours.
  • The decision in IBM’s Toronto Lab to split the DB2 development team into New Development and Continuing Engineering teams.
  • More generally, case studies of how real engineers made decisions on what components, vendors, libraries and languages to use for projects.

My New To-Do List

This book dramatically expanded my known to-do list for my thesis. In currently-planned order:

  1. Finish preliminary literature review reading.
  2. Write literature review.
  3. Rewrite research questions to be well-formed.
  4. Prune research questions that are not suitable for a case study.
  5. Write an short explanation of the case study research method and a defense of its validity, targeted at a software engineering researcher. That is, rewrite the above summary in a way suitable for inclusion in my thesis.
  6. Show the subset of the total case study method I intend to use for my own study. Justify.
  7. Write full case study design:
    1. Research questions
    2. Propositions
    3. Rival propositions
    4. Case study protocol (dependent on sources of evidence).
    5. Description of sources of evidence I intend to collect
    6. Description of analysis I intend to perform
    7. Chain of evidence for analysis and sources of evidence
  8. In parallel with (7), list 20-30 candidate cases. Solicit their interest in participation, filter as needed. I doubt I’ll have a lot of interest; I bet my case selection will be driven by whomever agrees to participate. Some ideal candidates off the top of my head:
    1. Unspace
    2. Shopify
    3. Mozilla
    4. Freshbooks
    5. Rypple
    6. Refresh
    7. Microsoft
    8. Google
    9. Red Hat
    10. EMC
    11. Big five banks
    12. Basie
    13. RIM
    14. Learnhub
  9. Seek preliminary IRB for aforementioned design (or real IRB if skipping 10-12)
  10. Lightweight preliminary pilot
  11. Revise protocol and design
  12. Full IRB for revised design
  13. Collect data
  14. Analyse data
  15. Revise and update literature review
  16. Edit

Freshly Updated Research Questions

June 8, 2009
Are DSL programs a significant target for maintenance effort?
    How does the maintenance of DSL programs compare to the maintenance of GPL programs?
        Does debugging of DSL programs consume significant effort?
            Does debugging of DSL programs take more or less effort than similar-sized GPL programs? Why?
                Does the nature of bugs in DSL programs differ from bugs in GPL programs?

                Do programmers form thorough models of DSL program execution?
                    Are these models correct?

                    Do programmers have difficulty forming cognitive models of alternative execution models?
                        How are programmers trained in specific DSLs?

                Do tool differences cause differences in difficulty?
                    Do GPL tools help with DSL maintenance?

                Do differences in size between DSL programs and GPL programs cause differences in difficulty?
                    How do we compare the size of a DSL program to the size of a GPL program?
                        Functionality?

                        Effort invested in creation?

                        Physical size, i.e. bytes/tokens/lines?

            Do programmers follow a similar process for debugging DSL and GPL programs?
                Do programmers perform slicing on DSL programs?

                Do programmers follow a known model of GPL debugging, e.g. Hale and Haworth 1991?

        Does change of DSL programs to meet new requirements consume significant effort?

    Why not?

Writing a thesis in public

June 8, 2009

I believe in openness, transparency and accountability. I believe in rapid iterations and honest feedback. I believe in backups.

Repository browser for my thesis!

hg clone http://www.arandonohue.com/hg/thesis

Or:  Atom RSS for changes

An unstable draft will be always be available here.

The system:

  • Mercurial repository browser
  • Francois Pitt’s LaTeX thesis template
  • TextMate TODO bundle
  • Paper notes are kept in LaTeX comments in the bibtex file
  • Copies of the papers are kept in an unversioned cited-papers/ directory
  • Post-commit hooks
    • Push the code to the web
    • Make a PDF and upload it

Defining “Domain-Specific Language”

June 8, 2009

Numerous definitions of Domain-Specific Language exist in the literature. Wu et al. define a DSL as follows:

A domain-specific language (DSL) is a programming language with concise syntax and rich semantics designed to solve problems in a particular domain.

Van Deursen:

A small, usually declarative, language expressive over the distinguishing characteristics of a set of programs in a particular problem domain.

Mernik et al.:

Domain-specific languages are languages tailored to a specific application domain.

Waite avoids the term DSL and instead focuses on declarative specifications, which he defines as descriptions

…which explicitly list properties that a solution must have.

Fowler:

…a limited form of computer language designed for a specific class of problems.

Wikipedia :

In software development, a domain-specific language (DSL) is a programming

language or specification language dedicated to a particular problem domain,

a particular problem representation technique, and/or a particular solution

technique.

I’ll assume the definition given by Mernik et al.

A note on terminology: I use the term Domain-Specific Language or DSL to refer

to a notation. The term Domain-Specific Language program or DSL program refers to a

specific instance of the language, i.e. a file that could be saved on a computer.

Confounding Effects of Domain-Specific Language Usage

June 6, 2009

One for the discussion section…

It is impossible to generalize research on domain-specific languages as much as we would like. Domain-specific languages are so numerous and vary so widely that any controlled laboratory experiment will miss important properties of the DSLs left unstudied. As for the study of DSLs in practice, we cannot claim that any result found is an intrinsic property of the domain-specific language approach. It is plausible, for example, that DSLs are only ever used where requirements are unusually precise or where the domain is easily modeled.

Draft Research Questions

June 5, 2009

I’ve been brainstorming research questions. Here goes:

Are DSL programs a significant target for maintenance effort?

If yes,

How does the maintenance of DSL programs compare to the maintenance of GPL programs?

Does change of DSL programs to meet new requirements consume significant effort?

Does debugging of DSL programs consume significant effort?

About the debugging process:

Do programmers follow a similar process for debugging DSL and GPL programs?

Do programmers perform slicing on DSL programs?

Do programmers follow a known model of GPL debugging, e.g. Hale and Haworth 1991?

About the cost of debugging:

Does debugging of DSL programs take more or less effort than similar-sized GPL programs? Why?

Do tool differences cause differences in difficulty?

Do differences in size between DSL programs and GPL programs cause differences in difficulty?

Here we can compare a

DSL program of comparable functionality.

or a

DSL program of comparable physical size, i.e. bytes/tokens/lines.

Do programmers form thorough models of DSL program execution?

Are these models correct?

Do programmers have difficulty forming cognitive models of alternative execution models?

How are programmers trained in specific DSLs?

I also have some draft interview questions, which I’ve related to the research questions. They’re even less developed, but you can see the graph:research and interview questions (click for larger size)

Draft Thesis Abstract and Introduction

June 2, 2009

I set out to write approximately 500 words of thesis-related content among other tasks today, which worked out roughly to being a draft abstract and introduction. I would describe it as heavily unedited, since I have trouble editing this kind of stuff while writing it.

Tell me if it sounds like something you’d like to read more 🙂 Remember, heavily unedited, so caveat emptor.

Abstract

Language-oriented programming research focuses on the techniques, tools and benefits of domain-specific language creation. However, since software maintenance is an important part of software projects, we should consider the maintainability of domain-specific language programs. These programs differ in many ways from their general-purpose language counterparts; we should not expect results applicable to one set to generalize to the other.

This thesis presents the results of a case study of maintenance of domain-specific language programs in industry. Results show that TODO Complete when results are gathered.

Introduction

Language oriented programming and the related subject of domain-specific languages have attracted attention from academics and industry practitioners.

Academic research has studied the nature and benefits of domain-specific languages. Several domain-specific languages are described in the literature. Researchers have explored methodologies and tools for constructing domain-specific languages. Several papers describe empirical evaluations of domain-specific languages. A few describe techniques for automatic creation of debuggers and other tools for domain-specific languages.

Some domain-specific languages are targeted at non-programmer domain experts. These languages overlap with the research in end-user programming.

Prominent businesses host tool-development projects to simplify the creation of domain-specific languages. Books and essays feature domain-specific languages.

To our knowledge, little attention has been paid to the maintainability of domain-specific language programs. As deliverables, they are subject to requirements change; as programs they are vulnerable to defects. That maintenance constitutes a large part of software effort is now common knowledge. Existing efforts to construct debuggers for domain-specific languages imply a need for debugging of domain-specific languages. Together this implies a gap in our understanding of the total value of domain-specific languages. This thesis aims to fill the gap.

Chapter 2 of this thesis surveys the field of domain-specific languages and language oriented programming. It provides a thorough standalone introduction to the state of the art in tools and knowledge, including pointers to other surveys and resources.

Chapter 3 surveys research in debugging, with an emphasis on cognitive processes, theories and other results relevant to domain-specific languages.

Chapter 4 poses and justifies specific research questions relevant to the maintainability of domain-specific language programs. These questions were informed by lightweight exploratory research; this work is detailed here.

Chapter 5 details the design of the main case study of this thesis. Chapter 6 presents its execution. The results are analyzed in Chapter 7 and discussed in Chapter 8.

Trawl for DSL Debugging Anecdotes…

May 29, 2009
I intend to send out the following message. I welcome suggestions on targets and editing.
I’m looking for some (serious) anecdotes describing debugging experiences of domain-specific language programs. In particular, I want to know about particularly thorny bugs which caused you lots of headaches. I’d like to know how you cracked the problem– what techniques/tools you used: did you ‘home in’ on the bug systematically, did the solution suddenly come to you in your sleep, etc. A VERY brief stream- of-consciousness reply (right now!) would be much much better than a carefully-worked-out story. I can then get back to you with further questions if necessary.
Motivation: For my Master’s research I will study DSL debugging. DSLs get lots of hype—from Paul Graham, Dave Thomas, Charles Simonyi, Sergey Dmitriev, the Ruby crowd and others—but debugging issues are rarely discussed. This trawl is  inspired by Marc Eisenstadt’s “Tales of Debugging from the Front Lines.” In fact the text above is ripped from his paper.
Thanks!