Governor's One-on-One Mentoring Program annual report, fy ...

              Governor’s One-on-One Volunteer Program
Annual Evaluation Report
March 2009
Special Provision: S.L. 2007-323
Section 18.4
Submitted by:
Department of Juvenile Justice and Delinquency Prevention
Linda Wheeler Hayes, Secretary 2
ANNUAL EVALUATION OF COMMUNITY PROGRAMS:
GOVERNOR’S ONE-ON-ONE VOLUNTEER PROGRAM
TABLE OF CONTENTS
Table of Contents…………………………………………………………………………………… ….. .. 2
Executive Summary……………………………………………………………………………………….. 3
Section 1: Description of Program, Services, and Recipients……………………………………………. 4
Section 2: Meaningful Measures of Program Performance…………………………………………….. 14
Appendix A: Organizations Funded In 2007-2008……………………………………………………… 24 ANNUAL EVALUATION OF COMMUNITY PROGRAMS:
GOVERNOR’S ONE-ON-ONE VOLUNTEER PROGRAM
EXECUTIVE SUMMARY
The Department of Juvenile Justice and Delinquency Prevention offers the 2007-2008 Governor’s One-on-One Annual Evaluation Report in accordance with House Bill 1473, Session Law 2007-323, Section 18.4, Annual Evaluation of Community Programs. This statutory provision requires that the Department evaluate the degree to which participation in the Governor’s One-on-One Volunteer Program has resulted in a reduction of court involvement among juveniles.
The report is organized into two sections to include a description of the program, services and recipients of services, and an analysis of the measures of program performance. In describing the program services and recipients in Section One, the Department awarded grant funds to 42 organizations which operated 45 programs to provide mentoring services in 46 counties during FY 2007-2008. A total of 1,152 youth were served in a one-on-one mentoring relationship with an adult/volunteer mentor during the fiscal year. Juvenile Court, school resource officers, and law enforcement were the primary referral sources with 734 or 65 percent of the 1,152 youth served being referred from those entities to programs. Twenty-eight percent were referred to the programs for problems identified as delinquent behavior (person, property, or victimless crimes), while 27 percent of youth committed status offenses or were considered ungovernable. Sixty-six percent of youth served were at-risk for involvement in the juvenile justice system.
Section Two of this report discusses the process and outcome performance measures as they relate to elements of effective mentoring practices to the achievement of Department and legislative goals and program objectives. Process and outcome evaluation serves as the basis for measuring the effectiveness of services during FY 2007-2008. The 45 programs reported a total of 1,152 youth matched with an adult volunteer/mentor for at least 12 months, spending a total of 70,233 hours in one-on-one mentoring. The programs held a total of 261 training sessions totaling an average of 621 hours of training per program for volunteer mentors.
During the fiscal year, the 45 programs offered a total of 324 structured group activities equating to an average of 7.5 group activities per program. A review of compliance monitoring data revealed that the 45 programs achieved a total of 499 new matches, exceeding minimum client standards by 49 percent.
Program outcomes were especially strong during FY 2007-2008 including (1) a 22.8 % reduction in delinquent complaints, (2) a 32.1 % reduction in adjudicated delinquent complaints, and (3) 43% of programs reporting that parental involvement among youth served had improved from the time of admission to the time of termination.
3
ANNUAL EVALUATION OF COMMUNITY PROGRAMS:
GOVERNOR’S ONE-ON-ONE VOLUNTEER PROGRAM
Description of Program, Services, and Recipients
The first section of the Governor’s One-on-One Volunteer Program1 annual evaluation report provides a description of the program to include information on the services provided and the recipients of services. Toward this end, the section begins by offering the historical contexts for the Governor’s One-on-One Volunteer Program. The funding structure and processes the Department of Juvenile Justice and Delinquency Prevention (hereinafter “Department”) utilize to administer the Governor’s One-on-One Volunteer Program funds are also discussed. A description of the Governor’s One-on-One Volunteer Program service delivery model and core standards required by the grant-funded programs is provided, and the programs for FY 2007-2008 are highlighted in terms of several demographic profiles including gender, race, age, legal status, referral reasons and sources of referral.
Historical Contexts for the Governor’s One-on-One Volunteer Program: Executive Order and Comprehensive Strategy
The Governor’s One-on-One Volunteer Program was created by Executive Order in 1982 by former Governor Jim Hunt. This Executive Order established funding for 34 local Governor’s One-on-One Volunteer Programs to be administered by the Department of Human Resources, Division of Youth Services, in order to promote the development of local programs that matched court-involved young people with local adult volunteers to help direct youth activities away from delinquency. On March 20, 1989, House Bill 729 was enacted to appropriate funds for the expansion of 4 new local programs to provide adult volunteer friends and role models for troubled young people2.
The Department has administered the Governor’s One-on-One Volunteer Program since 2000. The Department’s mission is to promote public safety and juvenile delinquency prevention, intervention, and treatment through the operation of a seamless, comprehensive juvenile justice system that provides the most effective services to youth and their families at the right time in the most appropriate settings.
In fulfilling this mission, the Department works to provide the state of North Carolina with a Comprehensive Strategy that helps prevent and reduce juvenile crime and delinquency. The Comprehensive Strategy (see Figure 1-1) is a system for responding proactively to juvenile delinquency and crime as well as a framework that provides the Department with a practical method for evaluating its juvenile justice and delinquency prevention programs against best practices. It focuses on strengthening families, promoting delinquency prevention, supporting core social institutions, intervening effectively and immediately when delinquent behavior
1 The Governor’s One-on-One Volunteer Program refers to the funding initiative as administered by the Department of Juvenile Justice and Delinquency Prevention. Governor’s One-on-One Volunteer programs refer to programs awarded grant funds by the Department for the provision of mentoring services.
2 House Bill 729, Session 1989.
4
occurs, and identifying and controlling the small group of serious, violent and chronic juvenile offenders in local communities.
Figure 1-1: Comprehensive Strategy NC’s Comprehensive Strategy for Juvenile DelinquencyProblem Behavior >NoncriminalMisbehavior >Delinquency >Serious, Violent, and Chronic OffendingPreventionTarget Population: At-Risk YouthPreventing youth from becoming delinquent by focusing preventionprograms on at-risk youthGraduated SanctionsTarget Population: Delinquent YouthImproving the juvenile justice system response to delinquent offenders through a system of graduated sanctions and a continuum of treatment alternatives>>>>>>Programs for All YouthPrograms for Youth at Greatest RiskImmediate InterventionIntermediate SanctionsCommunity ConfinementTraining SchoolsAftercare
At the front end of the continuum, delinquency and early intervention prevention programs are resources for preventing and reducing the onset of juvenile delinquency offsetting major risk factors in individual, family, peer, school and community domains. This prevention continuum includes mentoring as one of nine effective service categories for prevention and court supervised delinquency programs.3
Since its formation in 1982, the Governor’s One-on-One Volunteer Program has evolved as a statewide mentoring initiative serving at-risk4 and court-involved5 youth. In accordance with the
3 See Lipsey, M. W., Howell, J. C., & Tidd, S. T. (July 2007). The Standardized Program Evaluation Protocol (SPEP): A Practical Approach to Evaluating and Improving Juvenile Justice Programs in North Carolina, Final Evaluation Report. Nashville, TN: Vanderbilt University, Center for Evaluation Research and Methodology.
4 At–risk Youth - Youth possessing a wide range of personal, familial, social, or academic circumstances wherein the propensity of school failure or involvement in anti-social activities without intervention to reduce risk factors increases.
5 Court-Involved Youth – Adjudicated delinquent or undisciplined youth with a personal profile including delinquent behavior, truancy, an unstable home environment, low self-esteem, substance abuse, below grade level school performance, family income below or at the poverty level, physical or mental health problems, and learning disabilities.
5
mission of the Department and in alignment with the Comprehensive Strategy, the Governor’s One-on-One Volunteer Program has the following aims:
1.
Provide structure for court-involved youth through mentorship and court guidance;
2.
Prevent juvenile crime in local communities served by the program by mentoring youth who are at risk of delinquency;
3.
Recruit community volunteers to provide positive adult role models for youth;
4.
Improve youth school attendance and performance, character, social, and leadership skills;
5.
Reduce juvenile delinquency and undisciplined behavior in youth involved with juvenile court; and
6.
Increase county awareness of and involvement in the problems of youth served in the juvenile justice system
As a result of close coordination and collaboration between youth and their families, adult volunteer/mentors6, public and private human service agencies, and the juvenile court system, the Governor’s One-on-One Volunteer Program has served youth and families for more than twenty-six years, becoming the catalyst for a comprehensive network of community-based services available to at-risk and court-involved youth in North Carolina.
Description of the Program: Governor’s One-on-One Volunteer Program Administration and Funding Structure
In FY 2007-2008, the North Carolina General Assembly appropriated $1,645,545 million to the Department to fund Governor’s One-on-One grant programs for the provision of mentoring services to at-risk and court-involved youth. Proposals to provide these mentoring services were solicited by the Department through a Request for Proposals and competitive grant process.
Through the grant process, the Department awarded single grants in counties to 501(c)(3) private, non-profit organizations or components of city, county, and state governments to deliver a Governor’s One-on-One mentoring grant program. Grant programs were selected for funding based on factors which include, but were not limited to the applicant’s ability to:
•
demonstrate the need for mentoring services for youth ages 6-17 years old to include community risk factors and how the proposed program will offset the risk factors that place youth at risk for juvenile delinquency;
•
develop a comprehensive plan for the implementation of a structured mentoring program for at-risk and court-involved youth;
•
recruit, screen, train, and retain qualified adult volunteers/mentors;
•
utilize elements of effective mentoring practices and strategies;
6 Adult Volunteer/Mentor - A trained adult volunteer eighteen (18) years of age or older matched with an at-risk or court-involved youth(s) either in a traditional one-on-one or group setting. 6
•
collaborate with Juvenile Crime Prevention Councils, local law enforcement, school systems, juvenile court, and other youth serving agencies;
•
deliver structured, age-appropriate, guided growth activities to assist youth as they explore careers, increase academic success, address character and social development and build stronger relationships with family and community; and
•
demonstrate ability to manage state funds in accordance with principles of sound internal accounting controls.
The Governor’s One-on-One grant funds are restricted for the purpose of subsidizing personnel expenses, solely salaries and fringe benefits for key personnel responsible for directing and/or coordinating the program. Programs are responsible for seeking additional revenues to complete local program operational budget.
The Department awards grant funds across five funding classifications or levels of funding ranging from $16,500 per year to $66,000 per year. The current funding formula allows for a full-time grant program to receive up to $33,000 per year in Department funds. Programs classified as full-time are required to employ a full-time staff person in the role of program director or program coordinator. The formula for determining all other classifications of funding is derived using the full-time classification of $33,000 as the basis. For example, a program classified as “half-time” receives one-half ($16,500) of the full-time funding classification amount.
At each classification of funding, programs are responsible for maintaining compliance with client capacity requirements. Client capacity refers to the minimum number of new and continuing youth matches7each year. Figure 1-2 outlines the five funding classifications, the funds available, and the minimum requirements for the number of matches for each classification.
7 Youth Match – Adult Volunteer/Mentor and Youth in a one-to-one mentoring relationship 7
Figure 1-2: Grant Funding Classifications, Allocations, and Requirements
Funding Classification
Grant Award
Client Capacity
Number of Grant Funded Programs
Half-Time
$ 16,500
Eight (8) youth matches with four (4) new youth matches per year
16
Three-Quarter Time
$24,750
Twelve (12) youth matches with six (6) new youth matches per year
3
Full-Time
$ 33,000
Seventeen (17) youth matches with eight (8) new youth matches per year
20
Full-Time and One-Half
$ 49,500
Twenty-two (22) youth matches with twelve (12) new youth matches per year
1
Double Time
$66, 000
Thirty-four (34) youth matches with sixteen (16) new youth matches per year
5
In FY 2007-2008, the Department awarded funds to a total of 42 organizations for the operation of 45 programs that provided Governor’s One-on-One mentoring services in 46 counties.8 Appendix A further depicts the award of grant funds by organization, region and counties served. The 45 programs employed 41 full-time and 35 part-time employees.
Description of the Program: Service Delivery Model and Requirements
The Governor’s One-on-One Volunteer Program service delivery model is designed for programs to serve at-risk and court-involved youth ages 6 to 17 through the provision of traditional one-on-one mentoring services9. The traditional mentoring model involves the placement of a caring adult volunteer/mentor and at-risk or court-involved youth in a mentoring relationship for a period of one year wherein the adult volunteer/mentor meets with the youth on a regular basis for a minimum of 2 hours a week and/or 8 hours a month. During the meetings, the adult volunteer/mentor strives to accomplish set goals and objectives to support the development of a valued relationship, improved behavior, improved academic performance and attendance, and improved social development with his/her youth.
The Governor’s One-on-One Volunteer Program service delivery model includes minimum standards and operating requirements for the programs. A description of minimum core standards and requirements for service delivery follows. These core standards set a foundation for a discussion in Section Two of this report in assessing how the Governor’s One-on-One Volunteer Program service delivery model aligns with national research on effective mentoring practices.
Governor’s One-on-One Volunteer Program Core Standards
8 Three agencies were each awarded grant funding to operate programs in more than one county. One agency operated a program serving two counties at less than Half-time status.
9 Effective July 1, 2008, grant funded programs will be required to implement group mentoring as a component of service delivery. Group mentoring is the placement of a caring adult volunteer/mentor with a group of 4-8 at-risk and/or court-involved youth in a group setting wherein community service (service learning), social and life skills development, academic enrichment, and career exploration are provided through group facilitation. 8
Youth Admissions and Terminations Standards
Each Governor’s One-on-One program must adhere to admissions criteria for youth referred to the local program.
Sixty percent of youth referrals served by the programs must originate from local law enforcement, Juvenile Court Counselor, Youth Development Center social worker, and/ or
a school resource officer during any 12-month period. Forty percent of youth referrals served by the programs may originate from other referral sources such as parents/guardians, school guidance counselors, and social services agencies.
Programs must demonstrate that at least 80 percent of all terminations from the program were successful completions10 and met the one year match obligation.
Required Number of Youth Matches
Within each classification of funding, programs are responsible for maintaining compliance with client capacity requirements. Figure 1-3 below illustrates the minimum number of youth required to be served in ongoing and new matches per each funding classification.
Figure 1-3: Minimum Client Capacity Requirements for Grant-Funded Programs
Funding Classification
Client Capacity
On-Going Matches
New Matches
Half-Time
8
4
Three-Quarter Time
12
6
Full-Time
17
8
Full-Time and One-Half
22
12
Double Time
34
16
Volunteer Recruitment Screening and Training
The service delivery model requires that all programs utilize a screening and training process for all volunteers interested in serving as an adult volunteer/mentor to a youth. Volunteer applicants must complete a screening process that includes an application, reference letters, criminal history checks to include Department of Motor Vehicles and Sex Offender Registry, a face to face or personal interview, and verification of vehicular insurance. Programs may require applicants to also complete a drug screening if warranted by the program’s policies and procedures.
10 Successful Completion – High level of client participation in program activities and achievement of behavioral improvement goals. 9
Programs are also required to have a Volunteer Screening Committee as part of their local Board of Directors and/or Advisory Board. The role of the Volunteer Screening Committee is to provide support and guidance to the program director/coordinator when special circumstances arise and to ensure that programs base decisions on volunteer/mentor selection using established guidelines and protocols commensurate to the general hiring practices of their local organization.
Volunteer applicants must complete a minimum of 3 to 6 hours of pre-service training. Pre-service training is the official orientation session for new volunteers/mentors. The pre–service training must cover specific curriculum topics as outlined by the Department.
Volunteers/mentors who have served more than 3 years with the program must attend an updated training. The new training session must include Department and organizational policy changes, a review of emergency protocols, and procedures for reporting monthly contact with youth.
Volunteer Activities and Support Services
Support services and activities must be provided by each program at least 4 times annually to recognize, motivate, and retain volunteers/mentors. Each program must offer at least 2 group activities for the adult volunteer/mentor and youth matches during the fiscal year and a minimum of 2 in-service training sessions for the volunteers during the fiscal year.
Matching Volunteers
The Governor’s One-on-One Volunteer Program service delivery model requires that each program match adult volunteers/mentors with youth based on compatibility factors determined by the program that are beneficial to the youth. Matching an adult volunteer with a youth is individualized according to compatibility, race, gender, and personal preferences. Male volunteers/mentors are not to be matched with female youth; female volunteers/mentors may be matched with male youth according to the specific guidelines and/or by-laws of the program agency. Volunteers/mentors must be at least 18 years of age at the time he/she is matched with a youth.
Program Director Qualifications and Training
Each program is required to be staffed with a local program director and/or program coordinator who has a minimum of Bachelor’s degree in a human services related field or a four-year college degree in any field plus at least two years experience as a direct service professional in a child-serving agency. New program directors/coordinators of programs are required to complete the Basics of One-On-One Volunteer Administration training provided by the Department within the first year of employment. Program directors are also required to attend four regional meeting training sessions facilitated by the Department on a quarterly basis and other trainings as deemed necessary by the Department.
10
Program Director/Coordinator Contact
The program director/coordinator of each program must maintain contact with adult volunteers, parent(s)/ guardian, and youth participating in the program. The program director or coordinator must establish and maintain personal contact with each volunteer/mentor once a week for the first 6 weeks after the volunteer/mentor has been matched with a youth with contract being made once a month thereafter for the remainder of the one-year commitment.
Contact with the parent/guardian of the youth being served is made within the first 2 weeks after the match is made. Contact must be maintained every month thereafter for the remainder of the year’s commitment. Contact with each matched youth is also performed on a monthly basis.
The program director/coordinator must maintain a filing system documenting contact with adult volunteers, parent(s)/guardian, and youth. At minimum, contact notes with the volunteer/mentor, youth, and parent/guardian should include the date, name of the person initiating contact, detailed conversation points, and specific updates demonstrating the ability of the volunteer/mentor to be a positive role model.
Recipients of the Services: Governor’s One-on-One Grant Funded Programs
Governor’s One-on-One programs are designed to serve a diverse population of youth between the ages of 6 to 17 who are at-risk and/or court-involved. For purposes of this discussion, recipients of mentoring services are defined interchangeably as youth participants or youth served, and are comprised of youth who have been admitted to or terminated from the program. An admission to the program occurs when a youth is matched with an adult/volunteer mentor, thus becoming a youth served in the grant funded program. A termination from the program occurs when the youth completes the program and/or when the youth is withdrawn or removed from the grant funded program. Conducting an analysis of data in this manner is appropriate due to the minimum one year match relationship requirements. Admissions to and terminations from programs a fluid and the number of youth participants can account for youth that are being served by programs across fiscal years. For example, a youth may be admitted into the program in May and be served in the match relationship into the next fiscal year.
The Request for Proposals process requires that programs conduct a needs assessment of existing conditions for youth participants describing how community risk factors placing youth at-risk for juvenile delinquency will be offset by the implementation of Governor’s One-on-One mentoring services.
Based on a review of 2007 rankings of county tier designations depicting the economic well-being of counties throughout the state, thirty-seven or 80% of the 46 counties served in FY 2007-2008 were classified as either “most distressed” or “moderately distressed”. The remaining 20% or nine counties were among the least distressed counties in the state11.
11North Carolina Department of Commerce, January 2009, http://www.nccommerce.com/en/BusinessServices/LocateYourBusiness/WhyNC/Incentives/CountyTierDesignation
11
An analysis of regional data for the 45 programs providing mentoring services in 46 counties demonstrated that unemployment rates exceeded the overall state rate of 4.7 percent in FY 2007-200812. According to a survey conducted by U.S. Census Bureau, American Community Survey, North Carolina’s poverty rate of 14.7% was slightly higher than the national averag13
Additionally, 20% of children in the state live in poverty and 34% of children reside in single parent families14. These factors, coupled with poor academic performance of economically disadvantaged students on end of grade tests in reading and math, corroborate mentoring research. The need for mentoring services for at-risk and court-involved youth from single parent households and low socioeconomic status to be involved in a long-term positive relationships is vital to a youth’s overall physical, mental, academic, and social development15. The propensity for youth from rural counties and distressed communities to have increased rates of academic failure and live in poverty comprise a majority of the youth served by the programs.
Demographic profiles of youth served are represented by gender, age, juvenile justice involvement, source of referral and reason for referral. During FY 2007-2008, the programs served 1,152 youth participants in traditional one-to-one mentoring services in 46 counties.
Programs served a slightly higher number of females at 54 percent while males represented 46 percent. The racial composition of youth served represented a diversified population as 34 percent were White, 56 percent were African-American, and the remaining 14% were classified as Asian, Hispanic, Multiracial, Indian, or other descent. Of these youth, 45 percent were ages 10 through 13, 24 percent were between 14 and 15, and 21 percent of youth were between the ages of 7 and 9.
Sixty-six percent of youth served by programs were identified as at-risk. These youth have not been adjudicated delinquent or undisciplined but demonstrate many of the risk factors that are strong predictors for delinquent behavior. Exposure to risks such as family conflicts and disruption, community drug and alcohol use, academic failures, early and persistent behavior problems in school, and the presence of gangs correlate to high rates for juvenile delinquency.
Approximately 25 percent of youth served were identified as having contact with a Department court services district office and have been diverted into programming or have penetrated the juvenile justice system.
Of the 1,152 youth served during FY 2007-2008, 28 percent were referred to the programs for problems identified as delinquent behavior (person, property, or victimless crimes) while 27
12 Bureau of Labor Statistics, January 2009, http://www.bls.gov
13 U.S. Census Bureau, American Community Survey Calculations by Center on Budget and Policy Priorities, August 2008, www.cnn.org/pdf/2008/ACStotalpoverty.pdf.
14 Annie E. Casey Foundation , January 2009, http://www.aecf.org/MajorInitiatives/KIDSCOUNT.aspx http://www.kidscount/database center
15 Mentoring; A Promising Strategy for Youth Development, February 2002, http://www.childtrends.org/files/mentoringbrief2002.pdf.
12
percent of youth committed status offenses or were considered ungovernable as illustrated in Figure 1-3.
Figure 1-3: Youth Participants by Reason for Referral to Grant Funded Programs
FY 2009-2010
N=1,152 Youth Served
Source: DJJDP Client Tracking System
Reason for Referral At Admission
Youth Participants
Abused
1 %
Dependent
2%
Neglected
10%
Other
27%
Problem Behavior (Person Crime)
8%
Problem Behavior (Property Crime)
8%
Problem Behavior (Victimless Crime)
12%
Runaway
1%
Truancy
4%
Ungovernable
27%
As discussed in Section One of this report, a core standard of the Governor’s One-on-One Volunteer Program service delivery model is the requirement that 60 percent of the youth referrals are provided by juvenile court, school resource officers, or law enforcement while the remaining 40 percent of referrals are from other sources including, but not limited to, parent/guardians, mental health and social services agencies, or school personnel. Seven hundred and thirty-four or 64.6 percent were referred to the programs by juvenile court, school resource officers and/or law enforcement.
Summary
Section One of the Governor’s One-on-One Annual Evaluation Report provides a historical context for the establishment of the Governor’s One-on-One Volunteer Program and goals by which mentoring is identified as a needed prevention and intervention service for at-risk and court-involved youth. An overview of the administration and funding structure of the Governor’s One-on-One Volunteer Program is included and highlights that the Department awarded grant funds to 45 programs to provide mentoring services in 46 counties during FY 2007-2008.
The Governor’s One-on-One Volunteer Program service delivery model is detailed to include the minimum core standards for service delivery of grant funded programs and a description of the youth served during FY 2007-2008. A total of 1,152 youth were served in a one-on-one mentoring relationship with an adult/volunteer mentor. Juvenile court, school resource officers, and law enforcement were the primary referral sources as 734 or 64.6 percent of the 1,152 youth served were referred to programs. Twenty-eight percent were referred to the programs for
13
problems identified as delinquent behavior (person, property, or victimless crimes), while 27 percent of youth committed status offenses or were considered ungovernable.
ANNUAL EVALUATION OF COMMUNITY PROGRAMS:
GOVERNOR’S ONE-ON-ONE VOLUNTEER PROGRAM
Meaningful Measures of Program Performance
The second section of the Governor’s One-on-One Volunteer Program annual evaluation report concentrates on the meaningful measures of program performance and the degree to which the Governor’s One-on-One Volunteer Program is meeting these measures.
This section examines the effectiveness of the Governor’s One-on-One Volunteer programs funded in FY 2007-2008 in the following output and outcome measures of program performance:
(1)
Number of matches and total volunteer/mentor hours
(2)
Number of volunteer/mentor trainings and total hours
(3)
Number of structured activities delivered for mentor and youth
(4)
Compliance with meeting minimum standards
(5)
Reduction of delinquent complaints
(6)
Reduction of adjudicated delinquent complaints
(7)
Reduction in violations of community supervision
(8)
Increased parental accountability
Process Evaluation: Contexts for Measures of Program Effectiveness
The Department currently utilizes process evaluation as a primary approach by which to measure the effectiveness of the Governor’s One-on-One programs. Process evaluation focuses on what services were provided and how those services were delivered. Its purpose is to (1) describe how the programs were implemented; (2) document and analyze the implementation of these programs; and (3) assess whether strategies were implemented as planned and whether expected outputs were produced.
This discussion begins with a context for evaluating the output measures of program effectiveness for the Governor’s One-on-One programs. The standards and operational requirements which shape the service delivery requirements, activities, and processes for the Governor’s One-on-One programs are informed and supported by what national mentoring research deems as elements of effective practice. The National Mentoring Partnership16 highlights the following four program practices identified essential for strong and effective mentoring relationships:
16 National Mentoring Partnership is an organization that promotes, advocates and is a resource for mentors and mentoring initiatives nationwide
14
15
1.
Conducting reasonably intensive screening of potential mentors
2.
Making matches based on shared interests of the youth and mentor
3.
Providing more than six hours of training for mentors
4.
Offering post-match training and support
In addition, an extensive empirical evaluation of program practice effectiveness was carried out by a meta-analysis of fifty-five evaluations of mentoring programs17. This study found seven different mentoring program practices as predictive of strong positive effective on youth outcomes:
1.
Use of community-based mentoring (as opposed to school-based settings only)
2.
Procedures for systematic monitoring of program implementation
3.
Utilization of mentors with backgrounds in helping roles and professions
4.
Clearly established expectations for frequency of mentor-youth contact
5.
Ongoing (post-match) training for mentors
6.
Structured activities for mentors and youth
7.
Support for parent involvement
The Governor’s One-on-One Volunteer Program service delivery requirements are inclusive of these practices deemed effective by national mentoring research. Figure 2-1 provides a comparison of the Governor’s One-on-One service delivery requirements and core standards as outlined in Section One of this report to these elements of effective practice for mentoring programs.
17 DuBois, D.L., Holloway, B.E., Valentine, J.C., & Cooper, H. (2002). Effectiveness of mentoring programs for youth: A meta-analytic review. American Journal of Community Psychology, 30, 157-197. Figure 2-1: Governor’s One-on-One Volunteer Program Standards in Relation to Elements of Effective Practice
Governor’s One-on-One Volunteer Program Standards
Elements of Effective Practice
Intensive Screening of Potential Mentors
Making Matches Based on Shared Interests
Mentor Training and Support
Expectations for Frequency of Contact
Support for Parent Involvement
•
Written application updated once every 2 years.
•
4 Reference letters
•
A local and statewide criminal background check. Includes NC DMV and Sex Offender Registry
•
Applicants relocated to North Carolina for less than one (1) year must have a criminal background check conducted in previous state of residence.
•
Drug screenings when required by the grant program
•
Documented in-depth interview
•
Verification of a valid drivers license.
•
Verification of auto insurance
•
Grant programs must have Volunteer Screening Committee as part of the Board of Directors and/or Advisory Board.
•
Matching an adult volunteer with a child is individualized according to compatibility, race, gender, and personal preferences.
•
Male volunteers are not to be matched with female youth.
•
Female volunteers may be matched with male youth according to specific program guidelines and/or by-laws.
•
Volunteers must be at least 18 years of age at the time he/she is matched with a youth
•
Minimum of 3-6 hours of pre-service training.
•
Training to cover specific curriculum topics as outlined by DJJDP.
•
Volunteers having served more than 3 years must attend an updated training to include policy changes, a review of emergency protocols, and procedures for reporting monthly contact with youth.
•
Support services and activities provided at least 4 times annually to recognize, motivate, and retain volunteers.
•
Minimum of 2 group activities provided and documented for the adult volunteer and youth matches during the fiscal year.
•
Minimum of 2 in-service training sessions for the volunteers during the fiscal year.
•
Grant program personnel must establish and maintain personal contact with each volunteer once a week for the first 6weeks after the volunteer has been matched with a child.
•
Contact must be made once a month thereafter for the remainder of the one-year commitment.
•
A volunteer must commit to a one (1) year service agreement.
•
Each volunteer must maintain a minimum of 2 hours of contact per week and/or 8 hours per month with a youth
•
Grant program personnel must establish and maintain contact with a parent/guardian of child being served within the first 2 weeks after the match is made.
•
Contact must be maintained every month thereafter for the remainder of the year’s commitment.
16
Process Evaluation: Outputs in Relation to Elements of Effective Practice
In using a logic model framework and process evaluation as the primary basis for evaluation, the service delivery and requirements (processes) for which programs are accountable resulted in the achievement of several outputs or evidence that service delivery occurred. Included in this discussion are reports on the following four output measures of program effectiveness in relation to elements of effective practice:
(1)
Number, duration, and frequency of contact for mentor matches
(2)
Volunteer training
(3)
Support activities for mentor matches
(4)
Monitoring of programs: Effectiveness based on meeting minimum client capacity standards
Throughout this section, excerpts from youth perspectives about their experiences in mentoring relationships are provided to provide additional context on the effectiveness of the services provided by the programs.
Output Performance Measure (1): Number, Duration, and Frequency of Contact for Mentor Matches
National mentoring research has shown that the duration or length of the mentoring relationship is an indicator of positive outcomes for youth. Given the fact that development of trusting relationships builds over time, national mentoring research recommends that mentors and youth should meet regularly for at least 4 hours per month for at least nine months.18
During FY 2007-2008, the 45 programs reported a total of 1,152 youth matched with an adult volunteer/mentor for a period of at least 12 months.
The 1,152 volunteers/mentors spent a total of 70,233 hours in one-on-one mentoring relationships with youth during the twelve month period. This equates to an average of 5.1 hours per month per match relationship.
“AYC Photo Event: This is such a great confidence builder and relationship builder! I’ve seen all of the youth gain so much confidence from this event. They learned precious life lessons from the memorable National Geographic presentation, developed confidence in remembering what they learned and what a thrill to see the pictures they took displayed in public throughout the county! They will never forget this, and I believe this activity has made a huge difference in their lives. Also, the gifts that we received (cameras, picture albums & frames) provided many hours of creating memories together, for which Gennie and I are very grateful!.” Kathleen, Mentor, Ashe County
18 National Mentoring Partnership: How to Build a Successful Mentoring Program Using the Elements of Effective Practice.
17
Output Performance Measure (2): Volunteer Training for Mentor/Youth Matches
Programs are mandated to provide a minimum of three hours of training for adult volunteers/mentors. National research asserts that volunteer training is a strong predictor of positive youth outcomes. In FY 2007-2008, the 45 programs held a total of 261 pre-service trainings totaling an average of 621 hours of training per program for volunteers recruited, screened, and eligible for being matched in a one-to-one relationship with a youth. These programs also provided ongoing support of the mentoring relationships by offering a total of 115 in-service trainings for adult volunteers/mentors in a one-to-one relationship with a youth. At an average of 3 hours per training, this equates to 345 hours of in-service training per program
In-service training topics for volunteers/mentors consisted of CPR certification, community resources and poverty training, and keys to educational success for youth. The vast majority of programs also held recognition events to retain and motivate volunteers/mentors.
The combined number of pre-service and in-service training opportunities per program averaged 7.5 trainings per program. At an average of 3 hours per training, programs provided a total average of 22.5 hours of training for volunteers/mentors.
Saving Grace Farm Group Activity: “Our students learned to work with supporting staff and peers and grasp many important qualities. They learned how to control anxiety, frustration, and the program is a calming experience that helps them learn to control these feelings in their home and environment. It is tangible and applicable to everyday life. If you take these qualities of cooperation, leadership and responsibility into your homes, schools and community the results are very meaningful and powerful” Liz, Program Coordinator, Rowan County
Output Performance Measure (3) Structured Activities Mentor/Youth Matches
In addition to volunteer training, national research suggests that structured activities supports the match relationship and serves as a predictor of positive youth outcomes. Structured activities are known to enrich the one-on-one mentoring relationships, increase the knowledge base of adult and youth participants, and strengthen community collaborations.
During FY 2007-2008, the 45 programs incorporated group activities centered on developing key life coping skills that included conflict resolution, gang awareness, communication, cultural awareness, and etiquette skills. Substance abuse prevention topics were also reviewed and discussed, specifically methamphetamine use and its effect on families.
Health Camp- I learned how to power the work out machines and how to get a great work out in about 30 minutes and that it can be fun to work out and that is about it . All these events i have learned something that matters in the world ,no matter who you are and that meeting new people is not hard cause i meant 3 and did not have any trouble talking or getting along with them and that all these activities are a great thing to do as a group” Gennie, Ashe County meetee
There were several commonalities among programs regarding the incorporation of physical fitness in agency sponsored activities. Rope courses, bowling, swimming, golfing, basketball, fishing, and skating were offered to the matches. Service learning activities included community
18
gardening, community beautification projects, and roadside and beach clean-ups. Many youth experienced their first visit to a college and/or university as participants in a Governor’s One-on-One program. Programs also held “Back to School” activities and holiday celebrations for youth, families, and volunteers/mentors.
As a community-based mentoring initiative, the traditional mentoring model allows youth and volunteers/mentors to develop their own set of approved activities in which they can routinely participate in throughout the term of the match. Common activities for youth and volunteers/mentors include tutoring and homework help, outings to the library, movies, concerts, theatres, museums, and sporting events.
The Governor’s One-on-One Volunteer Program service delivery model requires that programs provide a minimum of 4 structured activities to support the match relationships during the 12 month period. In FY 2007-2008, the 45 programs offered a total of 324 structured group activities which equates to an average of 7.5 group activities per program.
“You are the best mentor and I’m thankful that your in my life. I know you don’t like listening to my teenage drama, but you do anyway, and your give me terrific advise. I also know that sometimes I do things to disappoint you, but your still always there for me. I feel like I can trust you with my feelings and talk to you about anything, and it’s because I know you won’t judge me. I appreciate that you are a great person and I trust you with my life. I really hope I can become half the person you are ” 16 year old mentee, Moore County
Output Performance Measure (4): Monitoring of Programs’ Effectiveness Based on Meeting Minimum Client Capacity Standards
Dubois, et. al. (2002)19 identified that procedures for systematic monitoring of program implementation constitute an element of effective mentoring practice. The Governor’s One-on-One programs are required to deliver mentoring services in accordance with the Department’s Governor’s One-on-One Volunteer Program Standards and Operational Requirements, contract terms and conditions, North Carolina General Statute 143C-6-21-23 and North Carolina Administrative Code. Department staff conduct routine programmatic and financial reviews of each program as well as utilize a structured compliance monitoring protocol to conduct on-site review of program operations at the local level.
Routine and programmatic reviews are conducted monthly through submission of monthly activity reports by the programs, and on-site reviews are conducted at least once by Department staff during the program’s three year grant cycle.
As introduced in Section One of this report, the Department awards grant funds across five funding classifications ranging from Half-Time to Double-Time classification. At each classification of funding, programs are responsible for maintaining compliance with client capacity requirements. Client capacity refers to the minimum number of new and continuing youth matches each year.
19 DuBois, D.L., Holloway, B.E., Valentine, J.C., & Cooper, H. (2002). Effectiveness of mentoring programs for youth: A meta-analytic review. American Journal of Community Psychology, 30, 157-197. 19
A core indicator of compliance for programs monitored by the Department is the degree to which the programs meet the client capacity standards for the number of new matches as required at each funding classification level.
Section One of this report introduced the minimum number of new matches required at each classification of funding. Using this information as a guide, the 45 programs were responsible for meeting a total minimum number of 334 new matches during FY 2007-2008. A review of compliance monitoring data demonstrates that the 45 programs achieved a total of 499 new matches during FY 2007-2008. Programs exceeded the minimum client capacity standards by 165 matches or 49 percent.
The Department offers multiple services and support for various programs across four regions of the state: Eastern, Central, Piedmont, and Western. The 45 programs are dispersed statewide and the overall effectiveness of programs regarding compliance with minimum client capacity requirements for new matches was exceeded by an average of 67.25 percent. Based on data collected by programs, the Western region exceeded client capacity by 74 percent, Piedmont region by 63 percent, Central region by 69 percent, and the Eastern region reported 63 percent.
Figure 2-2 illustrates a comparison of the effectiveness of the Governor’s One-on-One programs in terms of length/duration of the match relationship, frequency of contact, structured group activities, and volunteer training hours as output performance measures against minimum standards of effective practice as cited in national mentoring research literature20.
20 National mentoring research literature does not provide a minimum number of structured group activities required for positive youth outcomes. Therefore, the value for this output is defaulted to one (1) activity per mentoring program.
20
Figure 2-2: Comparison of Output Performance Measures between Elements of Effective Practice and Governor’s One-on-One Volunteer Programs
N=45 Grant-Funded Programs
Source: 2007-2008 Governor’s One-on-One Volunteer Report and DJJDP Client Tracking System
Frequency of Contact
Length of Match
Relationship
25
20
15
10
Output
5
0
5.1
7.5
22.5
9
4
12
6
1
Governor's One-on-One
Grant Funded Programs
National Standard
Structured Group Activities
Volunteer Training Hours
Outcome Evaluation: Meaningful Measures of Program Effectiveness - Reduction in Court Involvement Among Juveniles
Using a logic model framework once again lends support that mentoring services provided by the local programs correlated to positive outcomes. As required by Session Law 2007-323, Section 18.4, Annual Evaluation of Community Programs, the Department focuses its attention in evaluating the degree to which the participation in the Governor’s One-on-One Volunteer Program has resulted in a reduction of court involvement among juveniles as the outcome measure of program effectiveness.
Included in this discussion are reports on the following three additional outcome measures of program effectiveness related to reductions in court involvement among juveniles:
(1)
Reduction of Delinquent Complaints
(2)
Reduction of Adjudicated Delinquent Complaints
(3)
Increased Parental Accountability
For these analyses, the Department uses two data sources: the North Carolina Juvenile On-Line Information Network (NC-JOIN) and the Client Tracking Database. NC-JOIN is a secure web-based application that manages the Department’s confidential juvenile information regarding complaints, court actions, activities involving the supervision of juveniles under the jurisdiction of the Department as well as information regarding admissions and supervision of detailed juveniles and juveniles in Youth Development Centers. On a monthly basis, client tracking data are submitted as electronic files (diskettes and emailed data tables) from the programs to the Department where the data are then compiled and transferred to a central database. The data in
21
client tracking database are relied heavily upon by the Department staff in conducting its monitoring of client capacity requirements for programs.
Outcome Performance Measure (1): Reduction of Delinquent Complaints
An important outcome for all Department programs is the impact on public safety. As highlighted in Section One of this report, one goal of the Governor’s One-on-One Volunteer Program is to reduce juvenile delinquency and undisciplined behavior in youth involved with juvenile court. A logic model framework also presupposes that the inputs and processes involved with the mentoring programs will likely result in several outcomes including enhanced public safety and reductions in juvenile crime. One indicator utilized by the Department in measuring this outcome is the reduction of delinquent complaints. One would assume that if effective mentoring practices are implemented and program objectives are met, then youth and families would receive services that would ultimately result in a reduction of delinquent complaints.
To ensure that reductions in delinquent complaints were not due to the ‘aging-out’ of the sample of youth in the juvenile system, this analysis examined youth who were 15 and a half years of age or younger 6 months prior to admission and 6 months subsequent to termination from the Governor’s One-on-One programs. An admission occurs when the youth is matched with an adult volunteer/mentor. A termination occurs when a youth completes the grant funded program and/or is withdrawn or removed from the grant funded program.
Of the 507 terminations for FY 2007-2008, there were 79 total delinquent complaints six months prior to or on youth admission date to programs. There were a total of 61 delinquent complaints six months following termination or on the program completion date. This yielded a 22.8 percent reduction in delinquent complaints.21
Outcome Performance Measure (2): Reduction of Adjudicated Delinquent Complaints
The reduction of delinquent adjudications reflects the prevention of youth from becoming more deeply involved in the juvenile justice system. To ensure that reductions in subsequent adjudicated delinquent complaints were not due to the ‘aging-out’ of the sample of youth in the juvenile system, this analysis examined youth who were 15 and a half years of age or younger 6 months prior to admission to and 6 months subsequent to termination from the Governor’s One-on-One programs.
There were a total of 28 total adjudicated delinquent complaints six months prior to youth admission to the programs. There were a total of 19 adjudicated delinquent complaints six months following termination or program completion date. This reflected a 32.1 percent reduction in adjudicated delinquent complaints22.
21Data current as of January 30,2009.
22Data current as of January 30,2009. 22
Outcome Performance Measure (3): Increased Parental Accountability
The Governor’s One-on-One Volunteer Program service delivery model seeks to engage parents in the provision of services by way of personal contact with the parents during the duration of the youth match with the adult volunteers/mentors. Client Tracking also records the level of parental involvement during the program as an outcome measure.
In FY 2007-2008, there were a total of 507 youth terminated from the grant funded programs for which this outcome measure was reported. Of the 507 terminations, 43 percent of the programs reported that parental accountability (involvement) among youth served had improved from the time of admission to the time of termination in the programs while 29 percent reported that parental involvement was non-problematic at the time youth were admitted to the programs. Eighteen 18 percent of programs reported that the levels of parental involvement remained unchanged from the time of admission to the time of termination from the program.
Summary
Section Two of the Governor’s One-on-One Volunteer Program annual evaluation report discusses the process and outcome performance measures as they relate to legislative goals of the Governor’s One-on-One Volunteer Program. The Governor’s One-on-One programs are serving as effective intervention and prevention programs in keeping at risk youth from involvement in the juvenile justice system. The evidence of programs’ success ranges from comparisons of the service delivery model to elements of effective mentoring practices to the achievement of program objectives in terms of services provided.
This section demonstrates that the programs met or exceeded expectations in several elements of effective practice including duration of match relationships, frequency of contact between adult/volunteers and youth, number of structured group activities to support the match relationships, and the number of training offerings provided to support the adult volunteer/mentors in achieving the objectives of the match relationships.
Anecdotal evidence from youth served and staff within programs also indicated that many youth served in FY 2007-2008 demonstrated significant improvements in academic achievement, familial relationships, and community involvement. National mentoring research suggests that structured activities to support the match relationship and training for volunteers are strong predictors of positive youth outcomes. Finally, this section also demonstrated the effectiveness of Governor’s One-on-One programs in enhancing public safety through reductions in juvenile crime.
23
ANNUAL EVALUATION OF COMMUNITY PROGRAMS:
GOVERNOR’S ONE-ON-ONE VOLUNTEER PROGRAM
APPENDIX A
ORGANIZATIONS FUNDED IN 2007-2008
Region
County*
Grant Funding Classification
Grant Award Amount
Grantee
Central
Bladen
Full-Time
$33,000
NC Cooperative Extension Bladen County Center
Chatham
Half-Time
$16,500
Chatham Together!, Inc.
Columbus
Half-Time
$16,500
Columbus County Dream Center, Inc.
Cumberland
Full-Time
$33,000
Fayetteville Urban Ministry, Inc.
Durham
Double
$66,000
Durham Companions, Inc.
Harnett
Full-Time
$33,000
County of Harnett
Lee
Full-Time
$33,000
Lee County Department of Youth & Family Services
Orange
Full and Half-Time
$49,500
Volunteers for Youth, Inc
**Person
Half-Time
$8, 250
Roots and Wings of Person County, Inc.
Scotland
Full-Time
$33,000
Scots for Youth, Inc.
Wake
Full-Time
$33,000
Haven House Services, Inc.
Warren
Half-Time
$16,500
Warren County Youth Services Bureau
Eastern
Beaufort
Half-Time
$16,500
Beaufort County Board of Education
Chowan
Three Quarter-Time
$24,756
NC Cooperative Extension Chowan County Center
Dare
Full-Time
$33,000
County of Dare
***Edgecombe
Half-Time
$8,250
Harrison Family YMCA of Rocky Mount
Greene
Half-Time
16,500
Kinston/Lenoir County One-on-One, Inc.
Lenoir
Full-Time
$33,000
Kinston/Lenoir County One-on-One, Inc.
***Nash
Half-Time
$8,250
Harrison Family YMCA of Rocky Mount
New Hanover
Full-Time
$33,000
Brigade Boys and Girls Club, Inc.
Onslow
Full-Time
$33,000
Boys & Girls Club of Eastern North Carolina, Inc.
Pasquotank
Full-Time
$33,000
Friends of the Court, Inc.
Perquimans
Full-Time
$33,000
NC Cooperative Extension Perquimans County Center
Pitt
Full-Time
$33,000
Communities in School of Pitt County, Inc.
Wayne
Double-Time
$66,000
Smart Choices for Youth, Inc.
Wilson
Full-Time
$33,000
Juvenile Support Services, Inc.
* Forty-two organizations were awarded grant funds to operate mentoring programs in 46 counties.
(Greene and Lenoir county programs were operated by the same grantee organization)
(Nash and Edgecombe county programs were operated by the same grantee organization)
(Avery, Mitchell, and Watauga counties were operated by the same grantee organization)
** Grant award amended due to official start date of the Governor’s One-on-One Volunteer Program 24
25
*** Nash/Edgecombe Counties funded at less Half-Time status.
ANNUAL EVALUATION OF COMMUNITY PROGRAMS:
GOVERNOR’S ONE-ON-ONE VOLUNTEER PROGRAM
APPENDIX A
ORGANIZATIONS FUNDED IN 2007-2008
Continued
Region
County*
Grant Funding Classification
Grant Award Amount
Grantee
Piedmont
Anson
Three-Quarter-Time
$24,756
NC Cooperative Extension Anson County Center
Forsyth
Full-Time
$33,000
YWCA of Winston Salem
Guilford
Double-Time
$66,000
Big Brothers Big Sisters of Greater Greensboro
Moore
Full-Time
$33,000
Moore Buddies, Inc
Randolph
Half-Time
$16,500
Randolph-Asheboro YMCA
Rockingham
Half-Time
$16,500
Rockingham County Youth Services
Rowan
Full-Time
$33,000
Rowan Youth Services Bureau
Stanly
Half-Time
$16,500
Stanly County One-on-One Program, Inc.
Surry
Full-Time
$33,000
Surry County Board of Education/Mount Airy City Schools
Western
Alleghany
Half-Time
$16,500
Alleghany Connections, Inc.
Ashe
Three Quarter-Time
$24,756
NC Cooperative Extension Ashe County Center
Avery
Half-Time
$16,500
Western Youth Network, Inc.
Buncombe
Half-Time
$16,500
Big Brothers Big Sisters of Western North Carolina, Inc.
Caldwell
Full-Time
$33,000
Caldwell Friends, Inc.
Cleveland
Double-Time
$66,000
Communities in Schools of Cleveland County, Inc.
Gaston
Double-Time
$66,000
Boys & Girls Clubs of Greater Gaston, Inc.
Graham
Half-Time
$16,500
Graham County Board of Education
Mitchell
Half-Time
$16,500
Western Youth Network, Inc.
Watauga
Full-Time
$33,000
Western Youth Network, Inc.
Yancey
Half-Time
$16,500
NC Cooperative Extension Yancey County Center
.
* Forty-two organizations were awarded grant funds to operate mentoring programs in 46 counties.
(Greene and Lenoir county programs were operated by the same grantee organization)
(Nash and Edgecombe county programs were operated by the same grantee organization)
(Avery, Mitchell, and Watauga counties were operated by the same grantee organization)
** Grant award amended due to official start date of the Governor’s One-on-One Volunteer Program
*** Nash/Edgecombe Counties funded at less Half-Time status.