Measuring Non-Profit Impact

Effective impact measurement is essential for demonstrating value, improving operations, securing funding, and fulfilling mission accountability. This lesson covers comprehensive impact assessment frameworks and methodologies.

Impact Assessment Frameworks

1. Logic Models

Core Components

A Logic Model is a visual representation of the relationships among resources, activities, outputs, outcomes, and impact. The standard framework includes:

Logic Model Quality Criteria

• Clarity: Each component clearly defined; no ambiguity
• Sequentiality: Logical flow from resources to impact; each step connected to next
• Measurability: Outputs, outcomes, and impact can be measured (or have reasonable proxies)
• Attribution: Causal link between program and outcomes can be supported (or limitations honestly stated)
• Stakeholder-Informed: Developed with input from those served; reflects community reality

Common Logic Model Pitfalls

• Output-outcome confusion: Reporting counts (outputs) as changes (outcomes); e.g., "500 workshops held" vs. "participants gained knowledge"
• Overly broad impact: Claiming credit for long-term societal changes impossible to attribute to single program; e.g., "we reduced city-wide poverty" when program serves 50 families
• Unrealistic measurement: Measuring things that can't actually be measured with available resources; e.g., surveys too long; data collection burden causes respondent fatigue
• Missing reverse causality: Not considering that participants may have certain characteristics before program; e.g., families who self-select into program may already be more motivated

2. Theory of Change

Core Concept

Theory of Change (TOC) is a comprehensive description and illustration of how and why a desired change is expected to happen in a particular context. It addresses the "missing middle" between activities and outcomes.

TOC vs. Logic Model Comparison

Theory of Change Components

• Long-term goal: The ultimate change the initiative seeks to contribute to
• Preconditions: Intermediate outcomes that must occur for long-term goal to be achieved; sequenced logically
• Causal links: Explicit "if-then" statements connecting each level; supported by evidence or justified reasoning
• Assumptions: External factors that must hold true; risks; context dependencies; made explicit and testable
• Activities/outputs: What the initiative does; mapped to how they contribute to each precondition
• Context factors: External conditions that enhance or hinder change; not controlled by initiative but must be considered

Developing a Theory of Change: Process

1. Define long-term goal: Articulate the ultimate change; be specific; ground in stakeholder vision; ensure it's achievable within initiative scope
2. Identify preconditions: Work backwards from goal; what must be true; sequence logically; typically 3-6 preconditions at each level
3. Map causal links: For each "if-then"; support with evidence (research, data, precedents) or justify reasoning; be explicit
4. Identify assumptions: For each link and the overall pathway; distinguish between controllable and uncontrollable; test where possible
5. Map activities: Connect program activities to each causal link; illustrate how inputs lead to outputs which lead to outcomes
6. Validate with stakeholders: Present to those served; staff; funders; experts; gather feedback; revise as needed
7. Document and use: Keep as living document; use for evaluation design; resource allocation; strategy decisions; communicate to stakeholders

3. Social Return on Investment (SROI)

Core Concept

SROI is a methodology for measuring and communicating a broader concept of value that incorporates social, environmental, and economic outcomes. It calculates the "social return" per unit of investment.

SROI Calculation Steps

SROI Ratio Interpretation

• SROI > 1:1: Social value created; for every $1 invested, $1+ of social value generated
• SROI = 1:1: Break-even; social value equals investment (not counting intangibles; distribution effects)
• SROI < 1:1: Social value less than investment; may still be worthwhile for other reasons (mission, ethics, partial contribution)
• Typical Ranges by Sector:
  • Education: 2:1 to 10:1 (varies widely by level, population, metrics)
  • Health: 3:1 to 15:1 (depends on intervention; preventive care often higher; treatment variable)
  • Human Services: 1:1 to 5:1 (housing, food security, employment services variable)
  • Environment: 5:1 to 20:1+ (ecosystem services; climate mitigation often very high; biodiversity variable)

SROI Example (Hypothetical)

SROI Best Practices

• Involve stakeholders: In outcome identification; monetization choices; validation; not extractive; compensative where possible
• Be conservative: In monetization; avoid inflating; use ranges; acknowledge uncertainty; transparency is credibility
• Use multiple methods: SROI complemented by qualitative; mixed methods strengthen credibility; different perspectives; triangulation
• Report comprehensively: Not just ratio; include; what's included/excluded; assumptions; sensitivity analysis; stakeholder perspectives; limitations; timeline; context
• Iterate and improve: First SROI is learning; subsequent versions improve; refine assumptions; add stakeholders; update with new data

Mixed Methods Impact Assessment

Quantitative + Qualitative Integration

Mixed Methods Design Examples

Design A: Explanatory Sequential

1. Collect + analyze quantitative data (survey of 500 beneficiaries)
2. Use results to inform qualitative data collection (interviews with 20 selected participants to explain quantitative patterns)
3. Integrate results: quantitative patterns explained by qualitative insights; joint display; overall assessment

Design A: Embedded Quantitative + Qualitative

1. Primary focus: quantitative experiment (randomized control trial with 300 participants)
2. Embedded: qualitative case studies within the RCT sites; weekly observations; participant journals
3. Integrate: quantitative treatment+effects; qualitative mechanisms; combined display; comprehensive understanding

Design A: Concurrent Triangulation

1. Collect quantitative + qualitative data simultaneously (survey + focus groups with 30 participants; survey with 300 participants)
2. Integrate: compare; contrast; merge; connect; diverge; initiate; extend; challenge findings; overall assessment; validity checks

2. Stakeholder Analysis and Feedback Mechanisms

Stakeholder Identification

Feedback Mechanisms

3. Common Impact Measurement Pitfalls and How to Avoid Them

---

Practice Questions

Q1. Which impact assessment framework visually represents the relationship among resources, activities, outputs, outcomes, and impact?

• A) Theory of Change
• B) Logic Model
• C) SROI Calculation
• D) Stakeholder Analysis

Q2. What is the key distinction between a Logic Model and a Theory of Change?

• A) Logic Models are only for education programs; Theory of Change is for all sectors
• B) Logic Models focus on shorter-term, linear pathways; Theory of Change addresses broader systemic change with explicit assumptions and context
• C) Theory of Change is simpler; Logic Model more complex
• D) There is no meaningful distinction

Q3. An SROI ratio of 3:1 means:

• A) For every $3 invested, $1 of social value is created
• B) For every $1 invested, $3 of social value is created
• C) The program breaks even (social value = investment)
• D) The program loses money (social value < investment)

Q4. Which integration type uses quantitative data to show scale and qualitative data to show depth/context?

• A) Triangulation
• B) Complementarity
• C) Development
• D) Emancipatory

Q5. Which of the following is a common impact measurement pitfall where outputs are reported as outcomes?

• A) Impact Washing
• B) Outcome Confusion
• C) Attribution Error
• D) Data Collection Burden

---