Notes on “Case Study Research Design and Methods”

June 18, 2009

Chapter 1: Applicability of Case Studies and How to Do Them

Defn:

A case study is an empirical enquiry that investigates a contemporary phenomenon in depth and within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident. The case study inquiry copes with the technically distinctive situation in which there will be many more variables of interest than data points, and as one result relies on multiple sources of evidence, with data needing to converge in a triangulating fashion, and as another result benefits from the prior development of theoretical propositions to guide data collection and analysis.

Performing a case study entails:

  • Literature Review
  • Posing research questions
  • formal and explicit procedures
  • Protecting against threats to validity
  • Maintaining “chain of evidence”
  • Investigating and testing rival explanations
In advance of doing a case study:
  • Define the “case” being studied
  • Determine relevant data to be collected
  • What to do with data once collected

Decide: Exploratory, Descriptive or Explanatory research purposes?

How to choose a research method given a research question type:

  • How, Why: Experiment iff controllable events, else history or case study
  • Who, What, Where, How Many, How Much: Survey, Archival Analysis

Case study applicability: “How” or “Why” questions, in-depth explanation of social phenomena, contemporary events, little or no control over events.

Well-formed research questions = Who, What, When, Where, Why, How
Choose research method based on RQ’s, then reformulate RQ’s to match the research method.

Chapter 2: Research Design

Components of research design:

  • Study’s questions
  • Propositions
  • Units of analysis
  • Logic linking data to propositions
  • Criteria for interpreting the findings

Research Design Quality (items covered in detail below)

  • Construct Validity
    • Correct operational measure for concept being studied
    • Use multiple sources of evidence
    • Establish chain of evidence
    • Key informants review case study report drafts
  • Internal Validity
    • Causal relationships established
    • Pattern matching
    • Explanation building
    • Address rival explanations
    • Logic models
  • External Validity
    • Generalizing to what domain
    • Theory and replication logic
      • Literal replication across cases = same procedure -> same results
      • Theoretical replication across cases = different procedure -> predictably different results
  • Reliability
    • Reproducible
    • Case study protocol
    • Case study database

Keep in mind: Multiple cases != sampling. Different logical basis for generalization.

Chapter 3

A good case study investigator:

  • Ask good questions
  • Good listener
  • Adaptive and flexible
  • Firm grasp of issues being studied
  • Unbiased by preconceived notions

Human Subjects Protection:

  • Informed consent: Informed as to nature of study, formal record of volunteerism
  • Protection from harm, avoidance of deception
  • Protection of privacy and confidentiality
  • Protection of especially vulnerable groups

As pretraining, review:

  • Why the study is being done
  • What evidence is being sought
  • What variations can be anticipated
  • What would constitute supportive or contrary evidence for any proposition

Case study protocol: Per case instructions for the case study investigator.

  • Project goals, case study issues, readings
  • Field procedures:
    • Access & contact info for case site, procedural reminders
    • Sufficient resources when in the field
    • Procedure for calling for assistance
    • Clear schedule of data collection activities
    • Provisions for changes in mood and motivation of investigator
  • Case study questions to keep investigator on track, linked to potential sources of information
  • Levels of question targets:
    • Specific interviewees
    • Individual case <- focus of case study protocol
    • Pattern of findings across multiple cases
    • Entire study
    • Conclusions
  • Guide for case study report: outline, data formats, other documentation, bibliographical information

“Nor, in many instances, do case study reports end up in journals” 😦

Pilot case study

Refine data collection plans and procedures. Not a pretest. Formative. Preferably before final IRB approval. Usually a close, tolerant contact of an investigator.

Consider 20-30 candidate case studies, filter down to a few using well-defined criteria.

Chapter 4

Six common sources of evidence:

  1. Documents
  2. Archival records
  3. Interviews
  4. Direct observation
  5. Participant-observation
  6. Physical artifacts

Interviews

  • In-depth interview: Facts and opinions of key players
  • Focused interview: Short, mostly follow set of questions from case-study protocol. Corroborate evidence.
  • Formal survey: Quantitative data

Record if desired, but: Ask permission, ensure comfort of interviewee. Plan to transcribe recordings. Ensure facility with recorder.

Three principles of data collection

  1. Multiple sources of evidence
    1. Data triangulation
    2. Investigator triangulation
    3. Theory triangulation
    4. Methodological triangulation
  2. Create a case study database: Separation of collected data from reports.
    1. Case study notes
    2. Case study documents
    3. Tabular materials
    4. Narratives
  3. Maintain a chain of evidence
    1. Link report to evidence, evidence to protocol, protocol to study question.

Chapter 5: Analysis

Four General Analytic Strategies

  • Rely on theoretical propositions
  • Develop a case description
  • Use both qualitative and quantitative data
  • Examine rival explanations

Rival explanations:

  • Craft Rivals
    • Null Hypothesis: Observation result of chance
    • Threats to validity
    • Investigator bias
  • Real-Life Rivals
    • Direct rival: Other intervention had the effect
    • Commingled rival: Other interventions contributed to the result
    • Implementation rival: The implementation process, not the intervention, had the effect
    • Rival theory: A theory different from the original explains the result
    • Super rival: A force larger than (and including) the intervention accounts for the result
    • Societal rival: Social trends account for the results

Five analytic techniques

  • Pattern matching: Predicted outcomes (for all theories including rivals) vs. actual results
  • Explanation building: Iterative pattern matching
  • Time-series analysis: Comparing critical intervention timing with critical event timing. Building a chronology of events.
  • Logic Models: Causally-connected pattern matching.
  • Cross-case synthesis: Generalize commonalities from multiple cases

Chapter 6: Composing the report

Composing the case study report:

Target to the audience. Consider multiple reports if different audiences.

e.g. if thesis dissertation committee is audience, cite committee member’s work.

Four formats:

  • Single case report (narrative)
  • Multiple cases presented singly as narratives + chapter on cross-case analysis
  • Question and Answer format
  • Entire report is cross-case analysis. Chapters dedicated to separate cross-case issues.

Six compositional structures:

  • Linear-analytic (typical structure.) Problem, design, conclusions, implications.
  • Comparative. Repetitions of same case from different perspectives
  • Chronological. Written in order of real-world events.
  • Theory-building. Order dependent on theory.
  • Suspense. Inversion of linear-analytic. Shocking outcome presented first, followed by details of how it came to happen.
  • Unsequenced. No importance to sequence of chapters.

Start writing early. Literature review, case design -> bibliography, methodology.

After data collection, write descriptive data section (even before analysis).

Identify case study subjects unless ethical concerns preclude it. Anonymizing is lots of work.

Have informants review drafts.

Exemplary case studies:

  • Individual cases are of general public interest
  • Underlying issues are important
  • Case study is complete. Boundaries between phenomenon and context are clear. Investigator pursued all evidence, especially that would support rival explanations. Case study not cut short by resource or other constraints.
  • Consideration of alternative perspectives
  • Display sufficient evidence so that reader can come to own conclusions just from the evidence.
  • Composed in an engaging manner.

Overall Impressions

  • Yin spends a lot of effort justifying the validity of case studies, but I think it really comes down to the research questions. Case studies can help answer “how” or “why” questions but I don’t trust them for “whether” or other types of questions. Case studies seem to be used in policy  to evaluate programs and decisions. I don’t think this is valid. I think more solid empirical methods are needed to evaluate whether a program is achieving its goals. Case studies should be used to understand how and why (or why not).
  • Case studies are a lot of work. Yin says this a few times, actually, claiming that case studies are the most difficult of the research methods
  • I’m not sure how well my intended topic matches the case study method. Several of my questions seem better addressed by surveys.
  • The book was remarkably dry. Unnecessarily so in my opinion. It’s an instructional tutorial, and I don’t see why it needs to be written in a formal academic style.
  • I want to talk to someone who is good at case studies to find out what transferable skills I can expect to learn by pouring myself into this. The work/reward ratio for this thing seems pretty bad to me. I need to find some good extrinsic motivation.

Opportunities for Case Studies

Reading the book made me think of several possible interesting case study topics, one not software engineering.

  • The decision by the Athletic Centre to institute women’s only hours.
  • The decision in IBM’s Toronto Lab to split the DB2 development team into New Development and Continuing Engineering teams.
  • More generally, case studies of how real engineers made decisions on what components, vendors, libraries and languages to use for projects.

My New To-Do List

This book dramatically expanded my known to-do list for my thesis. In currently-planned order:

  1. Finish preliminary literature review reading.
  2. Write literature review.
  3. Rewrite research questions to be well-formed.
  4. Prune research questions that are not suitable for a case study.
  5. Write an short explanation of the case study research method and a defense of its validity, targeted at a software engineering researcher. That is, rewrite the above summary in a way suitable for inclusion in my thesis.
  6. Show the subset of the total case study method I intend to use for my own study. Justify.
  7. Write full case study design:
    1. Research questions
    2. Propositions
    3. Rival propositions
    4. Case study protocol (dependent on sources of evidence).
    5. Description of sources of evidence I intend to collect
    6. Description of analysis I intend to perform
    7. Chain of evidence for analysis and sources of evidence
  8. In parallel with (7), list 20-30 candidate cases. Solicit their interest in participation, filter as needed. I doubt I’ll have a lot of interest; I bet my case selection will be driven by whomever agrees to participate. Some ideal candidates off the top of my head:
    1. Unspace
    2. Shopify
    3. Mozilla
    4. Freshbooks
    5. Rypple
    6. Refresh
    7. Microsoft
    8. Google
    9. Red Hat
    10. EMC
    11. Big five banks
    12. Basie
    13. RIM
    14. Learnhub
  9. Seek preliminary IRB for aforementioned design (or real IRB if skipping 10-12)
  10. Lightweight preliminary pilot
  11. Revise protocol and design
  12. Full IRB for revised design
  13. Collect data
  14. Analyse data
  15. Revise and update literature review
  16. Edit
Advertisements

15 Responses to “Notes on “Case Study Research Design and Methods””

  1. Greg Wilson Says:

    What (in practice) is the difference between Exploratory, Descriptive or Explanatory purposes?

  2. Greg Wilson Says:

    Do you have to have a theory before you can start a case study? Or is the purpose of an exploratory case study to (help) develop a theory?

  3. Greg Wilson Says:

    Re the qualities of a good investigator (start of Chapter 3): I’m surprised “kind to small animals” isn’t on the list as well. How do you tell if you have these characteristics & are applying them? Most people think they’re good listeners, but are wrong.

  4. Greg Wilson Says:

    You quote “Nor, in many instances, do case study reports end up in journals” 😦 Does the book explain why not? And is that actually the case in empirical software engineering?

  5. Greg Wilson Says:

    Any suggestions on the design of a case study database? Templates you could clone?

  6. Greg Wilson Says:

    Re Chapter 5 (analysis): these all do seem to rely on having a theory that you’re testing. What do you do in the Exploratory case (assuming that word means what I think it means)?

  7. Greg Wilson Says:

    In your summary, what do you mean by a “whether” question?

  8. Greg Wilson Says:

    In your summary, you say, “Case studies seem to be used in policy to evaluate programs and decisions. I don’t think this is valid.” Yin and others clearly think it *is* valid — why have they reached a different conclusion from you?

  9. Greg Wilson Says:

    In your summary, you say, “Several of my questions seem better addressed by surveys.” Doesn’t a survey pre-suppose a strong theory?

  10. Greg Wilson Says:

    Regarding future plans, please don’t start writing up your literature review just yet — very possible that your focus will change again, so some work you would do now might be unnecessary.


  11. […] « Notes on “Case Study Research Design and Methods” […]

  12. Wogari Says:

    Can I USE THE SAME SUBJECTS FOR EXPERIMENT, INTERVIEW, INTERVIEW, AND FOCUS GROUP DISCUSSIONS IN ONE MIXED METHOD STUDY? i SUFFERED A LOT. THUS I NEED IMMEDIATE RESPONSE.
    THANK YOU BEFORE HAND.

  13. Cecilia Boakye Says:

    It is very educative. Can questionnaires be used in case studies?


  14. […] variety), they are often biased and don’t control for confounding factors. These aren’t meaningful case studies, but rather experience reports. They typically don’t have a central proposition defined in […]

  15. hari Says:

    good guidance


Comments are closed.

%d bloggers like this: