Microsoft PowerPoint - Jul_30_2015_Developing_a_Long_Term_Research_Agenda_to

Developing a Long-Term Research

Agenda

Learning objectives

By the end of this presentation, you will be able to:

• Recognize the importance of building a long-term

research agenda

• Identify the various stages in building evidence of a

program’s effectiveness

• Understand the key questions to consider prior to

developing a long-term research agenda for your

program

PART 1

Defining a long-term research agenda

What is a long-term research agenda?

• A long-term research agenda is a series of intentional or

planned program evaluations and research tools that

build towards addressing a research goal.

• Similar to a strategic plan, a research agenda generally

spans over several years.

• A research agenda is unique and should be tailored to

each individual program.

• A research agenda is a dynamic tool (i.e., a living

document) that should be revised/updated based on

new evidence, shifts in program direction, etc.

Long-term research agenda

Why is it important to have a long-term

research agenda?

• A research agenda sets clear goals for what program

stakeholders want or need to know about the program

years into the future

• A research agenda defines your destination, then

identifies the supporting steps that will get you there

• A research agenda continues to build evidence of

program effectiveness

• A research agenda demonstrates strategic investment

of funds in evaluation activities

Build a long-term research agenda

• What does a long-term research agenda look

like?

– What do we want to have learned 5 years from now? 10

years from now?

– Work backwards: define your destination, then name

the supporting steps that will get you there

– Each evaluation should build on what you learned

previously

– If you invest evaluation money strategically, scarce

resources can have a big impact

Example of a long-term research agenda

AmeriCorps program provides housing assistance for low-

income families.

Goal: Demonstrate that the program has a positive impact on beneficiaries via a randomized control

trial (RCT).

– Step 1: Collect program data, routinely, on family background characteristics and number of

families served. [1

cycle]

– Step 2: Process study: Is the program being implemented with fidelity to the model? [1

cycle]

– Step 3: Collect pre/post outcome data each year via annual survey. [2

cycle]

– Step 4: In addition to data collected from Steps 1&2, collect long-term outcomes data via follow-

up survey (1 year post- program) [2

cycle]

– Step 5: Demand for the program exceeds supply, so implement RCT by randomly assigning

families to receive housing assistance. Collect background data and survey data from all eligible

families. [3

cycle]

Example: Stages in a long-term research agenda

What to consider when developing a

long-term research agenda

• Program maturity

– How long the program has been in operation and its grant cycle

timing

• Existing evidence base

– Evidence that has already been generated on the program that the

long-term research agenda should build off

• Funder requirements and other stakeholder needs

– CNCS has specific evaluation requirements for its grantees and

those requirements should be embedded in a program’s long-term

research agenda

– Sometimes the same evaluation can meet the needs and

requirements of multiple funders

What to consider when developing a

long-term research agenda

• Long-term program goals

– A long-term research agenda should be designed to systematically

provide information that supports a program’s long-term strategic

goals

• Long-term research goals

– Programs should have long-term research goals that relate to

building evidence of effectiveness over time

• Evaluation budget

– The amount of the program’s funding base that will set aside for

evaluation activities each year or each grant cycle



Exercise Part I: Key considerations in developing a long-

term research agenda for your AmeriCorps program

Your AmeriCorps program

Program

maturity

Existing

evidence

Funder

requirements

Long‐term

program goals

Long‐term

research goals

Evaluation

budget

PART 2

Building Evidence of Effectiveness

Building evidence of effectiveness



  



 



 



 



 



Stage 1:

Identify a strong

program design

Stage 5:

Attain causal

evidence of

positive program

outcomes

Stage 3:

Assess program

outcomes

Stage 2:

Ensure eff ective

implementation

Stage 4:

Obtain evidence

of positive

program

outcomes

Evidence

Informed

Evidence

Based



 



 



 



      

     

 



Stage 1: What is the program model and

what is it supposed to do?

Stage 1:

Identify a

strong

program

design

Stage 5:

Attain causal

evidence of

positive

program

outcomes

Stage 3:

Assess

program

outcomes

Stage 2:

Ensure

effective

implementation

Stage 4:

Obtain

evidence of

positive

program

outcomes

Evidence

Informed

Evidence

Based

Stage 1:

Identify a

strong

program

design

Gather evidence to support program design (e.g.,

conduct a literature review and/or needs assessment)

Develop logic model

Pilot program



 



 



 



 



Stage 2: Is the program operating as

envisioned?

Stage 1:

Identify a

strong

program

design

Stage 5:

Attain causal

evidence of

positive

program

outcomes

Stage 3:

Assess

program

outcomes

Stage 2:

Ensure

effective

implementation

Stage 4:

Obtain

evidence of

positive

program

outcomes

Evidence

Informed

Evidence

Based

Stage 2:

Ensure

effective

implementation

Output performance measurement

Process evaluation



 



 



 



   

  

Stage 3: How well is the program

working?

Stage 1:

Identify a

strong

program

design

Stage 5:

Attain causal

evidence of

positive

program

outcomes

Stage 3:

Assess

program

outcomes

Stage 2:

Ensure

effective

implementation

Stage 4:

Obtain

evidence of

positive

program

outcomes

Evidence

Informed

Evidence

Based

Stage 3:

Assess

program

outcomes

Outcome performance measurement (e.g.,

non‐experimental evaluation design)



 



 



 



 



 

Stage 4: Is there evidence that the

program is producing expected results?

Stage 1:

Identify a

strong

program

design

Stage 5:

Attain causal

evidence of

positive

program

outcomes

Stage 3:

Assess

program

outcomes

Stage 2:

Ensure

effective

implementation

Stage 4:

Obtain

evidence of

positive

program

outcomes

Evidence

Informed

Evidence

Based

Stage 4:

Obtain

evidence of

positive

program

outcomes

Outcome evaluation



 



 



 



 



   

 

Stage 5: Is the program resulting in

change?

Stage 1:

Identify a

strong

program

design

Stage 5:

Attain causal

evidence of

positive

program

outcomes

Stage 3:

Assess

program

outcomes

Stage 2:

Ensure

effective

implementation

Stage 4:

Obtain

evidence of

positive

program

outcomes

Evidence

Informed

Evidence

Based

Stage 5:

Attain causal

evidence of

positive

program

outcomes

Impact evaluation (e.g., quasi‐experimental

design, experimental design)

 



 



  

 



  

 

  



  



  

 

 



Building evidence of effectiveness:

Stages 1-5

Output Performance

Measurement

Process evaluation

Outcome Performance

Measurement

Gather evidence that

supports program

design

Develop logic model

Pilot program

Outcome

Evaluation

Impact

Evaluation

Stage 1: Identify

a strong program

design

Stage 5: Attain

causal evidence of

positive program

outcomes

Stage 3: Assess

program

outcomes

Stage 2: Ensure

effective

implementation

Stage 4: Obtain

evidence of

positive program

outcomes

Evidence

Informed

Evidence

Based

  

 

  



  



  

 

 



Stage 1: Identify

a strong program

design

Stage 5: Attain

strong evidence of

positive program

outcomes

Stage 3: Assess

program

outcomes

Stage 2: Ensure

effective

implementation

Stage 4: Obtain

evidence of

positive program

outcomes

Exercise Part II: Building evidence of effectiveness

for your AmeriCorps program

Evidence

Informed

Evidence

Based

PART 3

Example Scenarios

  

 

  



  



  

 

 



Stage 1: Identify

a strong program

design

Stage 5: Attain

strong evidence of

positive program

outcomes

Stage 3: Assess

program

outcomes

Stage 2: Ensure

effective

implementation

Stage 4: Obtain

evidence of

positive program

outcomes

Scenario 1: Building a long-term research

agenda for a small, new program

Evidence

Informed

Evidence

Based

Scenario 1: Logic model for a small, new,

homelessness prevention program

Process

Outcomes

INPUTS ACTIVITIES OUTPUTS

Outcomes

Short-Term Medium-Term Long-Term

What we invest What we do Direct products from

program activities

Changes in knowledge,

skills, attitudes, opinions

Changes in behavior or

action that result from

participants’ new

knowledge

Meaningful changes,

often in their condition

or status in life

Funding Provide case

management

50 families (head of

households) received

Increase head of

households’ knowledge of

Increase head of

households’ adoption of

Reduce first-time

homelessness in the

4 FT Staff housing relocation

and stabilization

case management

services

responsible home owner

or tenant practices/skills

responsible

practices/skills

community

30 services

AmeriCorps 50 families (head of Increase head of Decrease likelihood of

members Provide educational

workshops

households) attended

workshops

households’ knowledge of

resources/services in

foreclosures and

evictions

Training community

              

      

                

           

          

             

          

         



         

     



      

          

     

           

Scenario 1: Key considerations in

developing a long-term research agenda

Small, new, homelessness prevention program

Program maturity AmeriCorps grantee with no prior years of program implementation and in its first

grant cycle. Operating in only one community site.

Existing evidence The program’s evidence falls in the first stage on the continuum as it has conducted a

needs assessment to determine which program activities are most critical to the

community it serves and a literature review to determine best practices for

implementing cor e service activities. No evaluations have been conducted on the

program.

Funder

requirements

Small grantees must conduct an internal or an external program evaluation by the end

of the second grant cycle. Small grantees are required to submit an evaluation report

AND an evaluation plan with their recompete application after completing two or more

three‐year cycles.

Long‐term

program goals

Achieve full program operation with efficiency and fidelity to the program’s central

model. Realize all expected program outcomes.

Long‐term

research goals

Generate data to facilitate program improvements and ensure an efficient, full

operation of the program’s service activities. Generate data on the program’s short‐

and medium‐term outcomes (see logic model).

Evaluation budget 10‐15% of the program’s annual funding has been set aside for evaluation activities.

             



         

        

       

    

       

    

        

      

   

    

        

      

  

      

  

Scenario 1: Long-term research agenda for a

small, new, homelessness prevention program

Evaluation activities Stage of

evidence

Grant

cycle

1 Develop a logic model and a detailed program implementation plan. 1: Identify strong

program design

Pre‐1

2 Create a data system to routinely collect performance measurement

data and background data on program beneficiaries and AmeriCorps

members. Program staff and members begin routine data collection

activities.

2: Ensure effective

implementation

Pre‐1 and

3 Develop a survey to collect short‐term outcome data, focusing on

beneficiaries knowledge of responsible homeowner/tenant

practices and knowledge of resources and services in the

community. Members administer pre/post surveys to program

beneficiaries and analyze data.

3: Assess program

outcomes

1 and 2

4 Conduct an internal process evaluation to determine if the program

is being implemented with fidelity to the central model. Make data‐

driven adjustments to the program’s implementation as needed.

2: Ensure effective

implementation

5 Conduct a non‐experimental outcome evaluation using an external

evaluator, measuring both short‐term and medium‐term outcomes.

3: Assess program

outcomes

  

 

  



  



  

 

 



Stage 1: Identify

a strong program

design

Stage 5: Attain

strong evidence of

positive program

outcomes

Stage 3: Assess

program

outcomes

Stage 2: Ensure

effective

implementation

Stage 4: Obtain

evidence of

positive program

outcomes

Scenario 2: Building a long-term research agenda

for a large, established AmeriCorps program

Evidence

Informed

Evidence

Based

Scenario 2: Example logic model for large,

established, environmental restoration program

INPUTS ACTIVITIES OUTPUTS

Outcomes

Short-Term Medium-Term Long-Term

What we

invest

What we do Direct products

from program

activities

Changes in knowledge,

skills, attitudes, opinions

Changes in behavior or

action that result from

participants’ new knowledge

Meaningful changes, often

in their condition or status

in life

Funding

Staff

200

AmeriCorps

State and

National

members

200 non-

AmeriCorps

volunteers

Conduct forest

enhancement and

restoration

Complete up-

keep activities to

enable native

plants to survive

Install

100,000

native trees

and shrubs on

public land

Remove 50%

of invasive

plant species

on 10 forest

sites

Increase diversity

and coverage of

native plant species

Reduce presence of

invasive plant

species

Improve habitat

spaces for wildlife

Increase survival rate

of native plant species

and wildlife

Maintain

conservation of

healthy, productive,

sustainable

ecosystems

Research



          

        

         

        

         

            

           

        

    



          

           

        



         

               

              

  

            

        

Scenario 2: Key considerations in

developing a long-term research agenda

Large, established environmental restoration program

Program

maturity

AmeriCorps grantee in its second three‐year AmeriCorps grant cycle. Already operating

in multiple sites and expects to add additional service sites.

Existing evidence Established data collection processes to collect performance measurement output and

outcome data. Conducted internal process evaluation yielding evidence that the

program is being implemented with fidelity in most service sites.

Funder

requirements

Large grantees must conduct an external impact evaluation by the end of the seco nd

grant cycle. Large grantees are required to submit an impact evaluation report AND an

evaluation plan for a future evaluation with their re‐compete application after

completing two or more three‐year cycles.

Long‐term

program goals

Achieve and maintain fidelity of program implementation across all existing sites and

any new service sites. Build stronger evidence of effectiveness to support future

requests for higher levels of funding to expand program operations.

Long‐term

research goals

Conduct an external impact evaluation to assess the program’s short‐ and medium‐term

outcomes. Four to six years is the minimum amount of time for program outcomes to be

realized . For this reason, the grantee will submit a request for an alternative evaluation

approach for timing considerations.

Evaluation

budget

15% of the grantee’s annual funding has been set aside for evaluation activities.

Grantee is seeking additional outside funding for the impact evaluation.

      

     

          

       

        

 



       

      

 



  

    

      

        

      

 



Scenario 2: Long-term research agenda for large,

established environmental restoration program

Evaluation activities Stage of

evidence

Grant

cycle

1 Conduct a quasi‐experimental design (QED) study using an

external evaluator, measuring all short‐ and medium‐term

outcomes over a six‐year time frame and relative to a matched

comparison group of sites (i.e., adjacent non‐serviced areas

that are similar to the pre‐restoration conditions at the

treatment sites).

5: Obtain

evidence of

positive

program

outcomes

2+3

2 Continue to collect and analyze output and outcome

performance measurement data on an annual basis.

3: Assess

program

outcomes

2, 3, 4, etc.

3 Conduct an internal process evaluation focusing on new

service sites to determine if the program’s new restoration

projects are being implemented with fidelity to the central

model. Make data‐driven adjustments to the program’s

implementation as needed.

2: Ensure

effective

implementati

PART 4

Exercise

  

 

  



  



  

 

 



Stage 1: Identify

a strong program

design

Stage 5: Attain

strong evidence of

positive program

outcomes

Stage 3: Assess

program

outcomes

Stage 2: Ensure

effective

implementation

Stage 4: Obtain

evidence of

positive program

outcomes

Exercise: Building evidence of effectiveness for

your AmeriCorps program

Evidence

Informed

Evidence

Based



Exercise Part I: Key considerations in developing a long-

term research agenda for your AmeriCorps program

Your AmeriCorps program

Program

maturity

Existing

evidence

Funder

requirements

Long‐term

program goals

Long‐term

research goals

Evaluation

budget

  

 

  



  



  

 

 



Stage 1: Identify

a strong program

design

Stage 5: Attain

strong evidence of

positive program

outcomes

Stage 3: Assess

program

outcomes

Stage 2: Ensure

effective

implementation

Stage 4: Obtain

evidence of

positive program

outcomes

Exercise Part II: Building evidence of effectiveness

for your AmeriCorps program

Evidence

Informed

Evidence

Based



Exercise Part III: Long-term research agenda

for your AmeriCorps program

Evaluation activities Stage of

evidence

Grant

cycle

Important points to remember

• A long-term research agenda is a developmental approach

to evaluation whereby evidence of effectiveness is built

over time.

• A long-term research agenda is unique and should be

tailored to fit each individual program.

• There is value to building evidence at all stages along the

continuum.

• A long-term research agenda should reflect an iterative

process where evidence is built gradually over time.

Key points to consider when developing

a long-term research agenda

Evaluation Resources Page

Resources

• CNCS’s Resources

– http://www.nationalservice.gov/resources/americorps/evaluation-

resources-americorps-state-national-grantees

• The American Evaluation Association

– http://www.eval.org

• The Evaluation Center

– http://www.wmich.edu/evalctr/

• National Science Foundation

– http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_4.pdf

Questions?