Source: PENNSYLVANIA STATE UNIVERSITY submitted to NRP
COST BENEFIT/EFFECTIVENESS ANALYSIS OF ARMY COMMUNITY SERVICES
Sponsoring Institution
National Institute of Food and Agriculture
Project Status
COMPLETE
Funding Source
Reporting Frequency
Annual
Accession No.
1008609
Grant No.
2015-48780-24275
Cumulative Award Amt.
$1,173,597.00
Proposal No.
2015-10541
Multistate No.
(N/A)
Project Start Date
Sep 1, 2015
Project End Date
Aug 31, 2020
Grant Year
2017
Program Code
[AACS]- Cost Benefit/Effectiveness Analysis of Army Community Service (ACS)
Recipient Organization
PENNSYLVANIA STATE UNIVERSITY
408 Old Main
UNIVERSITY PARK,PA 16802-1505
Performing Department
Children Youth & Families Cons
Non Technical Summary
Based on our work with the Army over the last three years, we understand that increasing evidencebased practice among its social service providers is a priority; specifically, the selection, implementation, evaluation, and sustainability of evidence-based programs/practices. Moreover, continuous program improvement requires on-going evaluations, including cost-benefit/costeffectiveness(CB/CE) evaluations.Therefore, this project's primary goal is to conduct a CB/CE analysis of ACS. Over the year, the Clearinghouse for Military Family Readiness at Penn State will conduct a full or partial CB/CE evaluation. Our objectives are to: (1) build a strong, functioning partnership with Army stakeholders; (2) conduct a rapid evaluability review of each ACS program components and assess the type of data available; (3) develop a comprehensive theory of change for the overall ACS system; and (4) determine and conduct the most appropriate methodological and analytic approach for a CB/CE evaluation.
Animal Health Component
100%
Research Effort Categories
Basic
0%
Applied
100%
Developmental
(N/A)
Classification

Knowledge Area (KA)Subject of Investigation (SOI)Field of Science (FOS)Percent
90161993070100%
Goals / Objectives
This project's primary goal is to conduct a CB/CE analysis of ACS. Over the year, the Clearinghouse for Military Family Readiness at Penn State will conduct a full or partial CB/CE evaluation. Our objectives are to: (1) build a strong, functioning partnership with Army stakeholders; (2) conduct a rapid evaluability review of each ACS program components and assess the type of data available; (3) develop a comprehensive theory of change for the overall ACS system; and (4) determine and conduct the most appropriate methodological and analytic approach for a CB/CE evaluation.
Project Methods
1) Build a strong, functioning partnership with Army Stakeholders including a full and comprehensive understanding of key strategic objectives. Given the complexity and intricacies of this project, we will build a strong working partnership with Army stakeholders.2) Conduct a rapid evaluability review of each ACS program component and assess the type of data currently available for a CB/CE evaluation of ACS. The review involves systematic examination, using our Rapid Evaluability Assessment Worksheet, of each program component to gain a clear understanding of each component's specific goals and objectives.3) Develop a comprehensive theory of change for the overall ACS system. Currently, the Army has not articulated a theory of change for ACS as a system. Given its scope of services, a theory of change model for the entire system is necessary as it explains why thespecific program components of ACS are important and needed as well as why and how the components work to produce synergistic change efficiently and with measurable benefit. To accomplish this objective, we will, in partnership with stakeholders, complete the following: (a) develop a common understanding of how ACS is presumed to have a positive impact ? an approach that has been successfully used in similar projects with our Military partners and (b) review all logic models for accuracy and to ensure they are current. If needed, we will revise models as appropriate.4)Determine and conduct the most appropriate methodological and analytic approach for a CB/CE evaluation of ACS. Once a comprehensive understanding of the data is completed, the research team will consult with our Army partners to discuss the mostappropriate methodological and analytic approach for the CB/CE analysis.

Progress 09/01/19 to 08/31/20

Outputs
Target Audience:The audience for this work is the Army Community Services headquarters staff. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?A webinar was conducted on the ERP outcome evaluation. How have the results been disseminated to communities of interest?A full report and summary sheet were developed for the FAP and ERP evaluations and provided to the ACS headquarters staff. The ERP products were distributed to the installation directorates for further disemination and the FAP products are currently under review by the installation management command leadership before dissemination to the larger Army community. Davenport, D., White, L. & Perkins, D.F. (2020). Army Community Service Employment Readiness Program outcome evaluation report. Clearinghouse for Military Family Readiness at Penn State. Davenport, D., White, L. & Perkins, D.F. (2020). Army Community Service Family Advocacy Program process evaluation report. Clearinghouse for Military Family Readiness at Penn State. White, L. Urusow, A. & Perkins, D.F. (2020). Army Community Service Employment Readiness Program outcome evaluation [Summary Sheet]. Clearinghouse for Military Family Readiness at Penn State. White, L. Urusow, A. & Perkins, D.F. (2020). Army Community Service Family Advocacy Program process valuation [Summary Sheet]. Clearinghouse for Military Family Readiness at Penn State. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? A process evaluation for the Family Advocacy Program (FAP) was conducted and an outcome evaluation of the Employment Readiness Program (ERP) was conducted.

Publications


    Progress 09/01/15 to 08/31/20

    Outputs
    Target Audience:The target audience was policy analysts and leadership within the Army related to investments in Army Community Service Programs. Specifically, we conducted a rapid evaluability review of each ACS program component and assess the type of data available for a CB/CE evaluation; developed a comprehensive theory of change for the overall ACS system; and determined and conducted the most appropriate methodological and analytic approach for a CB/CE evaluation of three ACS efforts (SOS, FAP Prevention, and ERP) Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided? Nothing Reported How have the results been disseminated to communities of interest?Recommendation where shared in the form of webinars and summary sheets with relevant polcy analysts, leadership, and frontline providers. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

    Impacts
    What was accomplished under these goals? 1. Strong partnership built and lead to addition work. 2. Completed evaluability on all 11 ACS programs 3. Comprehensive change throy developed for ACS and its components 4. Evaluation conduiced on SOS, ACV, and ERP. CBA on SOs, ACV and FAP Prevention

    Publications

    • Type: Other Status: Published Year Published: 2016 Citation: Jones, D., Crowley, D., White, L., & Perkins, D. F. (2016). An overview of costs of the Army Community Service Programs. Clearinghouse for Military Family Readiness at Penn State.
    • Type: Other Status: Published Year Published: 2018 Citation: Crowley, D., Jones, D., Davenport, K., White, L., & Perkins, D. F. (2018). Army Community Service Army Volunteer Corps Coordinator cost-benefit analysis report. Clearinghouse for Military Family Readiness at Penn State.
    • Type: Other Status: Published Year Published: 2020 Citation: Davenport, D., White, L. & Perkins, D. F. (2020). Army Community Service Family Advocacy Program process evaluation report. Clearinghouse for Military Family Readiness at Penn State.
    • Type: Other Status: Published Year Published: 2020 Citation: Davenport, D., White, L. & Perkins, D. F. (2020). Army Community Service Employment Readiness Program outcome evaluation report. Clearinghouse for Military Family Readiness at Penn State.
    • Type: Other Status: Published Year Published: 2017 Citation: Jones, D., Davenport, K., White, L, Crowley, D., & Perkins, D. F. (2017). Army Community Service Family Advocacy Program economic assessment report. Clearinghouse for Military Family Readiness at Penn State.
    • Type: Other Status: Published Year Published: 2017 Citation: Crowley, D., Jones, D., Butler, J., Davenport, K., White, L., & Perkins, D. F. (2017) Army Community Service Survivor Outreach Services cost-benefit analysis report. Clearinghouse for Military Family Readiness at Penn Stage.


    Progress 09/01/18 to 08/31/19

    Outputs
    Target Audience:The audience for this work is Army Community Services headquarters staff. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?We presented two webinars on the hypothetical cost-benefit analysis of FAP and the cost-benefit analysis Army Volunteer Corps. How have the results been disseminated to communities of interest?There have been a total of three reports to the Army Community Services headquarter staff. There have been briefings to the Army Chief of Staff. Each report included a one-page inforgraphic that was shared with the ACS staff. What do you plan to do during the next reporting period to accomplish the goals?We will complete the evaluability review of FAP anger managemet and parenting classes. The data collection will b completed and the evaluation report of the ERP will be disseminated.

    Impacts
    What was accomplished under these goals? We are completing evaluability assessment of Family Advocacy Program (FAP) anger management and parenting. We are collecting data on Employment Readiness Program (ERP). The data is longitudinal and involves three waves. Currently, we are initiating site visits to collect observational data on the ERP.

    Publications


      Progress 09/01/17 to 08/31/18

      Outputs
      Target Audience:We have worked with all relevant stakeholders to conduct an appropriate CB/CE analysis of ACS. Over the last 12 months, the Clearinghouse for Military Family Readiness at Penn State (Clearinghouse) has met its objectives to do the following: build a strong, functioning partnership with all Army stakeholders. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided? Nothing Reported How have the results been disseminated to communities of interest?All reports produced were reviewed by partners and disseminated to their extended network of collaborators. Summary Sheets (i.e., infographics) were created for all finalized reports to share with the public. What do you plan to do during the next reporting period to accomplish the goals?Our team is combining all of the listed products into a master report. Evaluation plans are being developed to guide evaluations for the FAP and ERP in 2018/2019.

      Impacts
      What was accomplished under these goals? All reports produced have been outlined in the "Other Products" section of this progress report. After the completion of the Cost Report, Site Visit Report, and the Telephonic Evaluability Assessment Report, our team collaboratively chose the following programs for continued investigation: 1) Survivor Outreach Services received a cost-benefit analysis, 2) Family Advocacy Program received an economic assessment due to a lack of robust evaluation data, 3) Army Volunteer Program received a robust cost-benefit analysis. In addition to the previously outlined major goals of the project, and within the scope of the current project, the Clearinghouse has also been asked by the Army partners to evaluate the Army Community Services Family Advocacy Program (FAP) and the Employment Readiness Program (ERP). Evaluation plans are currently being developed and evaluation activities are anticipated to begin in June 2018.

      Publications


        Progress 09/01/16 to 08/31/17

        Outputs
        Target Audience:We have worked with all relevant stakeholders to conduct an appropriate CB/CE analysis of ACS. Over the last 12 months, the Clearinghouse for Military Family Readiness at Penn State (Clearinghouse) has met its objectives to do the following: (1) build a strong, functioning partnership with all Army stakeholders; (2) conduct a rapid evaluability review of each ACS program component and assess the type of data available for a CB/CE evaluation; (3) develop a comprehensive theory of change for the overall ACS system; and (4) determine and conduct the most appropriate methodological and analytic approach for a CB/CE evaluation of ACS. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided? Nothing Reported How have the results been disseminated to communities of interest?All reports produced were reviewed by partners and disseminated to their extended network of collaborators. Summary Sheets (i.e., infographics) were created (or will be created) for all finalized reports to share with the public. What do you plan to do during the next reporting period to accomplish the goals?Our team will finalize the Army Volunteer Corp Coordinator partial cost-benefit analysis report and provide for review by partners. We will also begin program evaluations for FAP and ERP.

        Impacts
        What was accomplished under these goals? Note: all reports produced or under development have been outlined in the "Other Products" section of this progress report. After the completion of the Cost Report, Site Visit Report, and the Telephonic Evaluability Assessment Report, our team collaboratively chose the following programs for continued investigation: 1) Survivor Outreach Services received a cost-benefit analysis, 2) Family Advocacy Program received an economic assessment due to a lack of robust evaluation data, 3) Army Volunteer Corp Coordinator is currently under investigation for a partial cost-benefit analysis (report anticipated in October of 2017). In addition to the previously outlined major goals of the project, and within the scope of the current project, the Clearinghouse has also been asked by the Army partners to evaluate the Army Community Services Family Advocacy Program (FAO) and the Employment Readiness Program (ERP). An evaluation plan is being created and will be submitted for review by the partners in September. Work on the evaluation of these programs is anticipated to begin in November, 2017.

        Publications


          Progress 09/01/15 to 08/31/16

          Outputs
          Target Audience:Therefore, the primary goal of this project is to work with all relevant stakeholders to conduct an appropriate CB/CE analysis of ACS. Over the next 12 months, the Clearinghouse for Military Family Readiness at Penn State (Clearinghouse) will conduct a full or partial CB/CE evaluation of Army ACS. Our objectives are to do the following: (1) build a strong, functioning partnership with all Army stakeholders; (2) conduct a rapid evaluability review of each ACS program component and assess the type of data available for a CB/CE evaluation; (3) develop a comprehensive theory of change for the overall ACS system; and (4) determine and conduct the most appropriate methodological and analytic approach for a CB/CE evaluation of ACS. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided? Nothing Reported How have the results been disseminated to communities of interest?The reports are being reviewed by the partners. Once reports are completed, an infographic will be created to share with public. What do you plan to do during the next reporting period to accomplish the goals?We will complete Evaluability Assessment report for both parts. After the EAs have been completed for all eleven ACS programs, we will partner with the coordinating committee at ACSIM to identify the three programs most ready for either a CB/CE analysis, or an evaluation plan (EP). If there is robust, high-quality data available for a given program, a CB/CE analysis can be conducted. A CB and/or CE analysis will involve thorough mapping of outcome data availability. Where robust data is not currently available to perform a CB/CE analysis, we will take a multi-tiered approach that will provide a comprehensive evaluation plan (EP) for up to three programs. This plan will provide a step-by-step guide on how these programs can efficiently move to a full CB/CE analysis in the future. Again, due to limited resources in this phase of the project, only the three most suitable ACS programs will be chosen for either a CB/CE Analysis or an EP.

          Impacts
          What was accomplished under these goals? A cost analysis of all ACS programs has been drafted and feedback from partners has been solicited and recieved. By July, the cost analysis of ACS and its programs will be finalized and published. The first part of the Evaluability Assessment, telephonic feedback sessions, has been drafted and is currently under review with partners. The second part of the Evaluability Assessment, site visits, are currently being conducted with completion date as to end of August.

          Publications