Ten questions to better pilot programs : changes needed in pilot program design to produce clear, useful results

Education Fiscal Research Division Ten Questions to Better Pilot Programs Fiscal Brief August 8, 2008 Changes needed in pilot program design to produce clear , useful results. Executive Summary The majority of pilot programs in North Carolina have failed to produce clear evidence of success or failure. This has made it difficult for members to determine whether or not to expand or discontinue the programs. The General Assembly has expressed a strong desire to receive clear, objective evaluations of new programs. However, most pilot programs have been designed in ways that make quality evaluation impossible. The goal of this memo is to help policymakers avoid the pitfalls that have undermined past pilot programs. Policymakers should ask the following ten questions to ensure that new pilot programs will be able to provide clear results: 1 . What is the problem that needs solving? 2. How does the program address the identified problem? 3. What is the cost of the program if it is successful? 4. Is there a budget or spending plan? 5. What criteria will be used to determine the program’s success or failure? 6. What alternative programs/solutions might also address the problem? 7. Does the design of the program allow for meaningful evaluation? 8. Are there problems in the program design that will affect validity? 9. Is there sufficient time to observe effects? 10. Are there enough units of study to ensure statistical significance? With clearer results, policymakers will better be able to determine which programs work and which programs do not. Introduction Pilot programs are new initiatives implemented on a limited basis as a test or trial. Ideally, the small-scale pilot program will provide data showing whether or not the new program has potential to succeed on a larger scale, or whether it should be discontinued. The State of North Carolina has demonstrated an admirable willingness to try out new initiatives by funding new pilot programs. Unfortunately, policymakers have learned little from these efforts. North Carolina’s pilot programs have generally included provisions and funding for program assessment. Unfortunately, these pilot program assessments have often provided ambiguous results, making decisions on program expansion difficult for policymakers. The primary reason is that the pilot programs themselves have been designed in ways that inadvertently preclude meaningful assessment. Common problems with pilot programs include: • Unclear goals - what does it mean for a program to “work”? • Unclear criteria - what measurements will be used to determine if a program is successful? • No control group - results of the program are not compared against an independent group not affected by the pilot program; • Selection bias problems - sites that are in the program are systematically different than those that are not; • An inadequate timeframe in which to observe outcomes - some pilot programs have been discontinued before results can be observed; • An inadequate number of pilot sites - the number of sites is insufficient to produce meaningful data. The goal of this memo is to help policymakers avoid the pitfalls that have undennined past pilot programs.