Recipient Organization
PENNSYLVANIA STATE UNIVERSITY
408 Old Main
UNIVERSITY PARK,PA 16802-1505
Performing Department
(N/A)
Non Technical Summary
The Army Child, Youth, and School Services (CYSS) is a delivery system, which consists of four programs (i.e., Child Development; School Age; Youth Programs [YP]; and CYSS Liaison, Education, and Outreach) that intend to reduce potential conflict between mission readiness and parent responsibility. By requesting a strong inference evaluation of Army YP, CYSS intends to enhance its utility and effectiveness in supporting the healthy development of military youth. CYSS aims to help youth become high-functioning adults, which can strengthen Army family and Soldier readiness. To date, implementation of YP processes and evidence of intended YP outcomes or cost-effectiveness have not been assessed. Examining these components is required under DODI 1342.22. The Clearinghouse proposes to collaborate with the Army DCS, G9 to address gaps in knowledge regarding program functioning, outcomes, and improvement methods. The Clearinghouse will conduct a process evaluation, outcome evaluation, and cost-benefit analysis of the YP.
Animal Health Component
100%
Research Effort Categories
Basic
(N/A)
Applied
100%
Developmental
(N/A)
Goals / Objectives
The Clearinghouse's goalsare to do the following: (1) continue to build a strong, functioning partnership with all Army stakeholders; (2) develop a project plan that enables the Clearinghouse to conduct the most appropriate methodological and analytic approaches for the process and outcome evaluation (Note, an outcome evaluation will be conducted if the process evaluation determines consistency of program delivery across Army installations); and (3) conduct a cost-benefit analysis of YP (Note, the Clearinghouse will do this if the YP achieves stated outcomes).Specifically, the four lines of effort for this project are noted below and organized by task.Task 1: In collaboration with partners DCS, G9, the Clearinghouse will develop a project plan to achieve all tasks noted below.Task 2: The Clearinghouse will use a theory-driven, multi-tiered evaluation plan to examine implementation througha process evaluationto determine the extent to which programs are provided to audiences in a manner that is consistent with how they were intended to be implemented (Munro & Bloor, 2010;Weiss, 1997).Task 3: After the completion of the process evaluation in year 1, the findings will determine whether an outcome evaluation will be initiated. If the process evaluation finding indicates that there is an inconsistency and/or a lack of readiness for evaluation, the Clearinghouse will work with DCS, G9 to adjust the next phase and associated tasks.Task 4a: For the cost benefit analysis portion of this project, benefits-mapping procedures that were demonstrated in prior work (Belfield et al., 2015) will be employed during the development of outcome metrics in the outcome evaluation.Task 4b: If the findings from the outcome evaluation indicate YP effectiveness, acost-benefit analysiswill be conducted. However, if the outcome evaluation does not yield positive findings in terms of program effectiveness, then a comprehensive detailed plan of program refinement will be completed.?
Project Methods
Specifically, here are the methods for the four lines of effort for this project.Task 1: In collaboration with partners DCS, G9, the Clearinghouse will develop a project plan by engaging in organize communication and meeting to assure that project progress is discussed regularly and communicated appropriately.Task 2: The process evaluation will focus on the utilization of the program materials (e.g., educational and marketing) and program components as intended and will determine whether the target population is reached (Weiss, 1997). The Youth Program (YP) process evaluation will include a sample of installations that represent the Army holistically and will address implementation, including consistency and quality of delivery of services across installations, an assessment of the evaluability of program components, and coordination of YP services with other services.Task 3: If the process evaluation finding indicates that there is an inconsistency and/or a lack of readiness for evaluation, the Clearinghouse will work with DCS, G9 to adjust the next phase and associated tasks. The outcome evaluation will use information garnered from the initial phase of the process evaluation to identify and employ SMART outcomes (i.e., Specific, Measurable, Attainable, Relevant, and Time bound; Conzemius & O'Neill, 2005; Doran, 2008). Metrics from the outcome evaluation will be selected based on the alignment with the YP and the ability of metrics to be monetized for a potential cost benefit analysis. A cost benefit analysis will be completed if findings from the outcome evaluation indicate that the YP are effective in achieving stated goals. Thus, the outcome evaluation plan would directly inform the "benefit" element of the cost benefit analysis. The outcome evaluation will be done across select Army installations (i.e., a mixture of sites with varying sizes, commands, and locations) to ensure the representativeness of the result for the Army overall. The outcome-evaluation metrics will provide measurement of YP effectiveness in achieving outcomes of strategic relevance to the Army.Task 4a: For the cost benefit analysis portion of this project, benefits-mapping procedures that were demonstrated in prior work (Belfield et al., 2015) will be employed during the development of outcome metrics in the outcome evaluation. These metrics are identified as part of the outcome evaluation to understand how relevant outcomes should be evaluated/valued for the sake of economic analysis in terms of whether these outcomes can be directly converted to dollar amounts, can be indirectly valued, or cannot be monetized. The latter option may include important outcomes that could be targeted through program services or could be better suited for cost effectiveness analyses if measured. With DCS, G9 consultation, the Clearinghouse will produce a brief report that indicates the Clearinghouse's capability to pursue cost benefit analysis and specifies whether the findings from the outcome evaluation warrant cost benefit analysis. Cost benefit analysis can provide crucial information in an economic evaluation by determining net benefits for the program. Potential obstacles include a lack of directly measured costs, which could include the use of medical services or increased productivity. However, projection methods may be used to estimate costs based on precedence from prior studies, including cost-of-illness estimates and contingent valuation.Task 4b: If the findings from the outcome evaluation indicate YP effectiveness, a cost-benefit analysis will be conducted. However, if the outcome evaluation does not yield positive findings in terms of program effectiveness, then a comprehensive detailed plan of program refinement will be completed. The cost benefit analysis will involve two components: (1) cost analyses and (2) benefit mapping and analysis. Cost analyses of YP will be employed using an ingredients-based approach (Levin & McEwan, 2001). Clearinghouse researchers will map all necessary cost resources and inputs to implement the program. This mapping will be documented in figures that will be presented in a brief report and will include representation of the program ingredients as specified in original logic models of the program and inclusive of any new program adaptations. The Clearinghouse will consult with DCS, G9 and YP staff regarding the usable, available budget-resource data and the non-dollar resources that are critical to the functioning of the program (e.g., space requirements, participant time). Projected value and opportunity costs will be applied where costs were not explicit in budgets. Further, analyses of costs will consider variation across contextual and participant levels. Determinant upon these analyses, the Clearinghouse will provide ranges of costs, for each program, that represent variability across settings. Cost analyses will be crucial to understanding the degree to which necessary resources vary and how efficiencies can be introduced to reduce costs at the participant level. As part of this, the Clearinghouse will also investigate what infrastructure factors are important to consider for program efficiency and sustainability (Crowley et al., 2012). This includes an assessment of staff required to deliver services and consistency of program delivery across implementations and over time. The cost analyses will be completed first in order to provide a full assessment of the resources expended per participant for each program.?