Source: PENNSYLVANIA STATE UNIVERSITY submitted to NRP
ARMY YOUTH PROGRAM EVALUATION
Sponsoring Institution
National Institute of Food and Agriculture
Project Status
ACTIVE
Funding Source
OTHER GRANTS
Reporting Frequency
Annual
Accession No.
1031611
Grant No.
2023-48795-41332
Cumulative Award Amt.
$475,000.00
Proposal No.
2023-09103
Multistate No.
(N/A)
Project Start Date
Sep 1, 2023
Project End Date
Aug 31, 2026
Grant Year
2023
Program Code
[FF-L]- IYFC, Admin. Discretionary & Reim. Extension
Recipient Organization
PENNSYLVANIA STATE UNIVERSITY
408 Old Main
UNIVERSITY PARK,PA 16802-1505
Performing Department
(N/A)
Non Technical Summary
The Army Child, Youth, and School Services (CYSS) is a delivery system, which consists of four programs (i.e., Child Development; School Age; Youth Programs [YP]; and CYSS Liaison, Education, and Outreach) that intend to reduce potential conflict between mission readiness and parent responsibility. By requesting a strong inference evaluation of Army YP, CYSS intends to enhance its utility and effectiveness in supporting the healthy development of military youth. CYSS aims to help youth become high-functioning adults, which can strengthen Army family and Soldier readiness. To date, implementation of YP processes and evidence of intended YP outcomes or cost-effectiveness have not been assessed. Examining these components is required under DODI 1342.22. The Clearinghouse proposes to collaborate with the Army DCS, G9 to address gaps in knowledge regarding program functioning, outcomes, and improvement methods. The Clearinghouse will conduct a process evaluation, outcome evaluation, and cost-benefit analysis of the YP.
Animal Health Component
100%
Research Effort Categories
Basic
(N/A)
Applied
100%
Developmental
(N/A)
Classification

Knowledge Area (KA)Subject of Investigation (SOI)Field of Science (FOS)Percent
80660203070100%
Knowledge Area
806 - Youth Development;

Subject Of Investigation
6020 - The family and its members;

Field Of Science
3070 - Psychology;
Goals / Objectives
The Clearinghouse's goalsare to do the following: (1) continue to build a strong, functioning partnership with all Army stakeholders; (2) develop a project plan that enables the Clearinghouse to conduct the most appropriate methodological and analytic approaches for the process and outcome evaluation (Note, an outcome evaluation will be conducted if the process evaluation determines consistency of program delivery across Army installations); and (3) conduct a cost-benefit analysis of YP (Note, the Clearinghouse will do this if the YP achieves stated outcomes).Specifically, the four lines of effort for this project are noted below and organized by task.Task 1: In collaboration with partners DCS, G9, the Clearinghouse will develop a project plan to achieve all tasks noted below.Task 2: The Clearinghouse will use a theory-driven, multi-tiered evaluation plan to examine implementation througha process evaluationto determine the extent to which programs are provided to audiences in a manner that is consistent with how they were intended to be implemented (Munro & Bloor, 2010;Weiss, 1997).Task 3: After the completion of the process evaluation in year 1, the findings will determine whether an outcome evaluation will be initiated. If the process evaluation finding indicates that there is an inconsistency and/or a lack of readiness for evaluation, the Clearinghouse will work with DCS, G9 to adjust the next phase and associated tasks.Task 4a: For the cost benefit analysis portion of this project, benefits-mapping procedures that were demonstrated in prior work (Belfield et al., 2015) will be employed during the development of outcome metrics in the outcome evaluation.Task 4b: If the findings from the outcome evaluation indicate YP effectiveness, acost-benefit analysiswill be conducted. However, if the outcome evaluation does not yield positive findings in terms of program effectiveness, then a comprehensive detailed plan of program refinement will be completed.?
Project Methods
Specifically, here are the methods for the four lines of effort for this project.Task 1: In collaboration with partners DCS, G9, the Clearinghouse will develop a project plan by engaging in organize communication and meeting to assure that project progress is discussed regularly and communicated appropriately.Task 2: The process evaluation will focus on the utilization of the program materials (e.g., educational and marketing) and program components as intended and will determine whether the target population is reached (Weiss, 1997). The Youth Program (YP) process evaluation will include a sample of installations that represent the Army holistically and will address implementation, including consistency and quality of delivery of services across installations, an assessment of the evaluability of program components, and coordination of YP services with other services.Task 3: If the process evaluation finding indicates that there is an inconsistency and/or a lack of readiness for evaluation, the Clearinghouse will work with DCS, G9 to adjust the next phase and associated tasks. The outcome evaluation will use information garnered from the initial phase of the process evaluation to identify and employ SMART outcomes (i.e., Specific, Measurable, Attainable, Relevant, and Time bound; Conzemius & O'Neill, 2005; Doran, 2008). Metrics from the outcome evaluation will be selected based on the alignment with the YP and the ability of metrics to be monetized for a potential cost benefit analysis. A cost benefit analysis will be completed if findings from the outcome evaluation indicate that the YP are effective in achieving stated goals. Thus, the outcome evaluation plan would directly inform the "benefit" element of the cost benefit analysis. The outcome evaluation will be done across select Army installations (i.e., a mixture of sites with varying sizes, commands, and locations) to ensure the representativeness of the result for the Army overall. The outcome-evaluation metrics will provide measurement of YP effectiveness in achieving outcomes of strategic relevance to the Army.Task 4a: For the cost benefit analysis portion of this project, benefits-mapping procedures that were demonstrated in prior work (Belfield et al., 2015) will be employed during the development of outcome metrics in the outcome evaluation. These metrics are identified as part of the outcome evaluation to understand how relevant outcomes should be evaluated/valued for the sake of economic analysis in terms of whether these outcomes can be directly converted to dollar amounts, can be indirectly valued, or cannot be monetized. The latter option may include important outcomes that could be targeted through program services or could be better suited for cost effectiveness analyses if measured. With DCS, G9 consultation, the Clearinghouse will produce a brief report that indicates the Clearinghouse's capability to pursue cost benefit analysis and specifies whether the findings from the outcome evaluation warrant cost benefit analysis. Cost benefit analysis can provide crucial information in an economic evaluation by determining net benefits for the program. Potential obstacles include a lack of directly measured costs, which could include the use of medical services or increased productivity. However, projection methods may be used to estimate costs based on precedence from prior studies, including cost-of-illness estimates and contingent valuation.Task 4b: If the findings from the outcome evaluation indicate YP effectiveness, a cost-benefit analysis will be conducted. However, if the outcome evaluation does not yield positive findings in terms of program effectiveness, then a comprehensive detailed plan of program refinement will be completed. The cost benefit analysis will involve two components: (1) cost analyses and (2) benefit mapping and analysis. Cost analyses of YP will be employed using an ingredients-based approach (Levin & McEwan, 2001). Clearinghouse researchers will map all necessary cost resources and inputs to implement the program. This mapping will be documented in figures that will be presented in a brief report and will include representation of the program ingredients as specified in original logic models of the program and inclusive of any new program adaptations. The Clearinghouse will consult with DCS, G9 and YP staff regarding the usable, available budget-resource data and the non-dollar resources that are critical to the functioning of the program (e.g., space requirements, participant time). Projected value and opportunity costs will be applied where costs were not explicit in budgets. Further, analyses of costs will consider variation across contextual and participant levels. Determinant upon these analyses, the Clearinghouse will provide ranges of costs, for each program, that represent variability across settings. Cost analyses will be crucial to understanding the degree to which necessary resources vary and how efficiencies can be introduced to reduce costs at the participant level. As part of this, the Clearinghouse will also investigate what infrastructure factors are important to consider for program efficiency and sustainability (Crowley et al., 2012). This includes an assessment of staff required to deliver services and consistency of program delivery across implementations and over time. The cost analyses will be completed first in order to provide a full assessment of the resources expended per participant for each program.?

Progress 09/01/24 to 08/31/25

Outputs
Target Audience:The target audience being reached our the Army Child and Youth Program Headquarters staff (including G9). Ultimately, the findings of this evaluation will influence Army Youth Programming. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided? Nothing Reported How have the results been disseminated to communities of interest?The results from the process evaluation have been submit to the headquarters staff and DCS-G9 in the form a process evaluation report.We are currently creating a summary sheet for the Army field staff of Children and Youth Programs. What do you plan to do during the next reporting period to accomplish the goals?We will complete the data collection of outcomes and then provide an outcome evaluation report. If there are significant outcomes, we will complete a cost-benefit analysis report.

Impacts
What was accomplished under these goals? As required by goal one, we developed, with significant feedback from our partners, a project plan. For goal 2, we have completed all data collection for the process evaluation. The report was completed on July 2, 2025 which is year 2. We were delayed due to waiting for Army approval for site visits. The process evaluation included systematic environmental scans of YP facilities, programmatic observations, and interviews of YP stakeholders at six geographic locations. These locations included eight YCs that represented different Installation Directorates (IDs), facility sizes, geographic regions (e.g., OCONUS vs. CONUS, rural vs. urban), characteristics, and configurations (e.g., proximity to middle and/or high schools). The YCs were chosen by DCS, G9, approved by Installation Management Command (IMCOM), and accepted by the Clearinghouse. The Clearinghouse requested interviews with a variety of stakeholders, including YP management and staff, local, installation-level CYS leadership, parents/guardians, and organizational supports. Interviewees were selected based on availability by YP management (e.g., CYS Coordinators and YP Directors) at each YC. A total of 86 people were interviewed. The findings from this evaluability assessment indicate that the YCs are staffed by passionate teams who are dedicated to supporting Service members and families by providing high-quality care to youth ages 11 to 18 years. YP services (i.e., core program areas) are, in general, being provided as directed within the regulations and in partnership with organizational supports like school liaison officers and Military and Family Life Counselors. Each YC was unique, and many varied in operational dimensions (e.g., facility design, staffing numbers, and supplies); however, consistency in delivery of YP services across installations was found in terms of each YC teams' commitment to the following nine features of quality programming: 1. Physical Safety (e.g., youth participants are not afraid of being physically harmed by peers or adults while in YP) 2. Psychological Safety (e.g., youth participants are not afraid of being bullied, demeaned, or punished/humiliated for taking risks or making mistakes while in YP) 3. Supportive Relationships (e.g., YP and CYS staff show they care or provide encouragement and emotional help to youth participants) 4. Opportunities to Belong (e.g., creating an environment that promotes a sense of connection or belonging among youth participants) 5. Positive Social Norms (e.g., youth participants are expected to behave respectfully or act with personal integrity while in YP) 6. Appropriate Structure for Youth (e.g., youth participants experience adult support of youth initiated activities, a detailed delivery schedule, smooth transitions between activities, clear and consistent expectations, and age-appropriate monitoring) 7. Efficacy and Mattering (e.g., opportunities, like volunteer and service activities, help youth participants feel valued and capable and give them a sense that they can make a difference) 8. Opportunities for Skill Building (e.g., youth participants experience skill learning and practice opportunities; speakers, field trips, and resources for project-based art, crafts, sports, or activities are provided) 9. Integration of Family, School, and Community Efforts (e.g., families are invited to attend YP events or activities, consistent communications with youth participants' school, community interactions [e.g., community service], installation interactions) The interviews and observations did illuminate some clear opportunities for improvement. There was convergence across the visited YCs in the feedback shared and observed practices. A summary of the main themes is listed below. § A lack of community awareness of YP exists, which may hinder youth recruitment. § YCs experience operational challenges (e.g., supply purchasing delays) that appear to originate from cumbersome bureaucratic requirements. § A de-prioritization of the YP by installation leadership and local, installation-level CYS leadership exists at many YCs, and this situation impedes program quality and diminishes morale. § YP staff retention needs to be improved in order to promote youth-adult relationship consistency, which can be a protective factor for youth. § A refined training model could be developed to help YC teams better achieve the goals for YP as noted in the regulations and directives. The model could include enhanced lesson planning and implementation of creative youth engagement strategies. § YP management and staff collect and use limited data; thus, they are unable to measure outcomes, communicate impact, and engage fully in continuous quality improvement. The findings point to several current successes the Army and installation-level YCs could build on to continue to improve YP services. Seven recommendations follow that can address frequent themes that emerged in interviews and observations. These recommendations are for Headquarters of the Army (HQDA) DCS, G-9 and IMCOM, G-9. Additional details regarding the data that informed each recommendation and specific action items are included in the last section of this report. 1. Proceed with an outcome evaluation of YP. In addition, empower YP management and staff to communicate the impact of YP. 2. Provide marketing resources that are implemented consistently and remove barriers to raise parental and youth awareness of YP. 3. Modernize training and develop a training infrastructure that focuses on building YP teams' mastery of YP regulations. 4. Develop a database of evidence-informed lesson plans that are designed to fit the context of YP and maximize youth interest, learning, and engagement. 5. Build a technical assistance infrastructure that Army leadership can use to directly communicate with YP Directors and Assistant Directors. 6. Promote consistent use of funding and simplify the supply acquisition and facility repair/upgrade processes. 7. Enact strategies to promote the retention of YP staff to strengthen adult-youth relationships. This could include elevating Child and Youth Program Assistants positions and creating an upward mobility pathway for YP staff. For goal 3, we have co-developed an outcome evaluation and will implement all required paperwork in the Summer of 2025. For goal 4, whether a cost-benefit analysis will be completed will be determined by the outcome evaluation. If significant outcomes are discovered, a cost-benefit analysis will be completed.

Publications


    Progress 09/01/23 to 08/31/24

    Outputs
    Target Audience:We have launched the first phase of the Army Youth Program Evaluation, namely the evaluability study. As such we have primarily been work with Army Youth Program headquarter staff and installation-level staff. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?We are currently hosting orientation training for all Army Youth Program staff (N=1200). The purpose of the training is to increase their understanding of the evaluability study and its components. Moreover, we also present the options in terms of the way forward for the outcome evaluation, which is dependenton the process evaluationresults. How have the results been disseminated to communities of interest?There are no results yet as the process evaluation launch on July 1, 2024. What do you plan to do during the next reporting period to accomplish the goals?We will complete Task 2, the process evaluation, this coming year. The results from that will enable us to select the most suitable option for task 3 and 4.

    Impacts
    What was accomplished under these goals? Task 1 has been completed in that a working plan of action has been co-developed with DCS, G9, and Army Youth Program Headquarters staff. Task 2 has been launched, and we are currently collecting data from an ecological scan of youth programming through an international-level staff survey. In addition, we have begun the process of establishing the logistics for the site visits that are part of the process evaluation. Tasks 3 and 4 have not begun as we need to complete task 2 to inform the way forward for those tasks.

    Publications