Progress 11/01/22 to 10/31/23

Outputs
Target Audience:The primary youth audience of the evaluation is teen attendees of the USDA Secretary's National 4-H Conference. A total of 288 youth delegates attended National 4-H Conference in 2023 and increase of 34% over 2023. During registration, 63% listed Female, 34% listed Male, 1% indicated another gender, and 2% did not respond. The majority of delegates identified as White (58%), 14% as Black, 6% Hispanic, 6% Asian, 9% Native American, 6% two or more races, and 1% Pacific Islander. A total of 56 LGUs sent youth in 2023 which represents half of all Land Grants. Of those delegations, 44 were from 1862 institutions, 10 were from 1890 institutions, 2 were from 1994 institutions. In addition, seven military service branches sent a conference delegation. Age was normally distributed across ages 14-19 with a mean of 16.6 years. The conference attracted mostly long-term 4-H members, with 76% having five or more years in the program and just 12% having one year or less. Changes/Problems:Actual FTEs for this reporting period are 3.4% (i.e. 3.4% total for all staff) or 0.034 FTE under Participants. The form rounds to nearest tenth, so box for "nothing to report" must be checked to submit the report. What opportunities for training and professional development has the project provided? Nothing Reported How have the results been disseminated to communities of interest?2023 National 4-H Conference Evaluation Report final report provided to and accepted by USDA NIFA for award. Details framework for evaluation, methodology, summative and formative findings, and recommendations. 2023 National 4-H Conference Evaluation Summary was provided to and accepted by USDA NIFA. The 2023 Summary is posted on the National 4-H Conference Resources webpage. The evaluation team was invited by NIFA to share highlights from the 2023 Evaluation Report at the National 4-H Conference information session in November 2023.The recording and slides were posted on the National 4-H Conference Resource site and distributed via email in December 2023. The NIFA-created information session recording can be accessed https://youtu.be/2VbSN7qeaFo?si=E3HH6NDspu2q-x46 What do you plan to do during the next reporting period to accomplish the goals?Action steps in Year 3 will continue successful strategies and expand as needed to address any changes in Conference goals or outcomes. In addition, evaluation instruments (or edited versions) may be utilized for evaluation of the 2024 National 4-H Conference providing an existing framework to start 2024 from. Changes to National 4-H Conference organization, content and implementation made because of the evaluation results and recommendations benefit USDA NIFA, as well as LGU 4-H programs and future delegates at National 4-H Conference. The Year 3 actions steps toward the goals and objectives of this project will include: Meet with the 2024 Conference planning leads to discuss the 2023 report. Share a summary of the year-one findings on the National 4-H Conference information session in November 2023. Confirm any changes to the goals and outcomes of the Conference. Revise the Theory of Change and evaluation plan as needed. Participate in bi-monthly National 4-H Conference planning team zoom meetings. Revise and develop evaluation tools as needed. Attend the National 4-H Conference in April 2024 and implement the evaluation plan. Analyze the data and develop a 2024 comprehensive report and summary report.

Impacts
What was accomplished under these goals? The 2023 comprehensive evaluation report contained qualitative and quantitative assessments for the 2023 National 4-H Conference. Comparisons were made to data in the 2022 when merited. Requested additions for the 2023 plan included items on hospitality and the career fair. Evaluation findings and recommendations will be used by USDA NIFA and the Planning Committee to improve delegate experiences overall and ensure each experience on the agenda uniquely supports the overall goals of the Conference. The report verified that, overall, Conference is a quality youth development experience in which youth of diverse backgrounds can be successful. New recommendations were made where programming could be strengthened to better align with the logic model. USDA NIFA, LGU 4-H programs and delegates at the next Conference are the most immediate beneficiaries. Objective 1: Conduct an annual end-of-event/program survey that assesses the quality of the program for promoting positive youth development and youths' engagement in learning. A post-survey included items focused on the quality of the delegate experience. Positive Youth Development (PYD) occurs in context, and the chosen quality measures are known to increase PYD outcomes. Items in "Belonging" were related to whether delegates felt physically and socially safe with the other youth and adults at Conference. "Agency" questions focus on whether Conference allowed delegates to experience independence and the mentoring appropriate in a youth-adult partnership. In "Contribution" items delegates reported how Conference helped them to give back to themselves, other delegates, and their home communities. Quality measures are overall positive and encouraging. In Belonging measures, 89% or better of answers were aligned with a positive indication of quality. The lowest measure indicated 11% of delegates reported feeling left out all or most of the time. Agency measures responses indicate delegates had some room for independence - 93.5% did or usually figured things out for themselves, and 98.1% said they usually got to do things they liked. Contribution measures are quite strong; the most positive answer getting more than 78% response in all areas. Adjustments made to the schedule were positively received, and efforts to allow more youth voice and choice were recognized.While the quality measures were strong overall, a good goal would be to raise the yes (or no in reverse coded items) to 80% or higher and reduce the still positive, but weaker response rate. Objective 2: Conduct an annual end-of-program survey to measure learning outcomes for the program, determined in consolation with NIFA. The youth delegate post-survey was designed to assess the impact of the Conference experience on the outcomes of capacity for leadership, civic engagement and an increased level of agency/empowerment to participate in solving complex issues. Response rates of 85-90% indicate delegates intend to seek out civic engagement opportunities, feel a sense of responsibility to solve problems, and feel empowered to do so. The post-survey also included 11 items from the 4-H Common Measures Social Skills Assessment, measuring aspects of personal development and self-awareness that reflect skills needed to make effective group decisions. Mean responses ranged from a low of 3.4 to a high of 3.7 on Likert-type scale of 1 to 4. The highest means are for items related to working with people different from oneself. The lowest are for items reflecting empathy - managing one's own feelings and being thoughtful of others' feelings. These results are nearly identical to the 2022 means and show the same patterns. It appears youth attending understand diversity is important and value other perspectives. However, they are still learning how to apply knowledge and attitudes to behaviors. Objective 3: Conduct potential additional annual youth evaluations as appropriated, such as participatory evaluation with youth and/or follow-up survey(s) after the program concludes. Additional data on youth experience was collected through a daily "rate the day" survey released each evening, providing data about engagement in each session and attitudes about Conference. Roundtable sessions, Monument Tours, and the Dance were the highest rated activities by youth while the motivational speaker, the USDA Flag Raising and the break activities were less engaging. The most used words to describe each day were "fun" and "exciting." Focus groups of 19 Collegiate Facilitators and the four Youth Leadership Team members were conducted the last day. They had meaningful feedback about their experience, observations of the youth delegates, and reflections on the planning process. Suggestions for improvement were incorporated into the final report. A post-survey of LGU contacts was used to evaluate the experience of the LGU in recruitment, registration and participation, and showed great dedication to the program, contributed ideas for registration and logistics, and contained rich anecdotal information about impacts on delegates. A post-survey of federal partners was disseminated but no responses were collected. Objective 4: Utilize 4-H Common Measures for data collection. 11 items from the 4-H Common Measures Social Skills Assessment were in the post-survey, measuring aspects of personal development and self-awareness that reflect skills needed to make effective group decisions. Findings are under Objective 2. Objective 5: Provide appropriate data analysis. Quantitative data came from registration reports, daily surveys, and the delegate post-survey. Demographics, session ratings, and survey items using Likert-type scales were analyzed with descriptive statistics such as means and rankings. Comparisons of survey responses were made between subgroups using Chi-Square Goodness of Fit Tests. There were no significant differences in youth Belonging, Agency, or Contribution found between racial/ethnic groups, genders, or length of time in 4-H. Retrospective pre/post survey questions were analyzed with a paired T-test. Delegates reported a highly significant increase in their interest in Civic Engagement due to their conference participation. Qualitative data was used to support the quantitative data, for suggestions for improvement and exemplary quotes that demonstrate the impact and importance of the conference. Objective 6: Complete an annual program evaluation report. The 2023 Evaluation Report was completed then submitted to and accepted by NIFA. The report includes all data collected, relevant analysis, and section of conclusions and recommendations. Findings include overall strong measures of youth learning, sense of belonging, and feelings of efficacy. Each program element aligned with the overall program goals, although some elements were more effective and engaging than others. The Roundtable groups, which are the core of the conference, are the strongest element. Data show that delegates gained new knowledge, changed attitudes (particularly around being exposed to new ideas and opinions) and they intend to apply their skills in their local communities. A two-page Evaluation Summary highlighting delegate demographics, notable impacts, and key recommendations was completed, submitted to and accepted by NIFA. The 2023 summary is posted on the National 4-H Conference Resources webpage - see Products. Objective 7: Provide recommendations for the succeeding year's Conference based on evaluation results. Of the 40 suggestions for improvement or change presented in the 2022 Evaluation Report, 31 were meaningfully addressed for the 2023 conference. The 2023 Evaluation Report included 39 specific suggestions for change or improvement in the Conference. These included topics such as logistics and meals, communication during the conference, schedule refinements, facilitator support, and which program elements were in need of strengthening.

Publications

• Type: Other Status: Other Year Published: 2022 Citation: Hein, Wendy and Lingard, Jill (2022). Information Session for the 2023 National 4-H Conference: Evaluation Slides. Oregon State University and University of Nebraska-Lincoln. Slides of key findings were shared during the NIFA presentation to LGU contacts. 68 attendees, slides posted publicly online by NIFA. NIFA embedded slides in a PowerPoint presentation at https://www.nifa.usda.gov/sites/default/files/2023-12/2024%20National%204-H%20Conference%20Information%20Session%20PPT_remediated_0.pdf
• Type: Other Status: Other Year Published: 2023 Citation: Hein, Wendy and Lingard, Jill (2023). 2023 National 4-H Conference Evaluation Summary. Oregon State University and University of Nebraska-Lincoln. Details summary evaluation findings and recommendations. Summary Report provided to NIFA. Posted at https://www.nifa.usda.gov/sites/default/files/2023-11/2023%20National%204-H%20Conference%20Evaluation%20Summary-508.pdf
• Type: Other Status: Other Year Published: 2023 Citation: Hein, Wendy and Lingard, Jill (2023). 2023 National 4-H Conference Evaluation Report. Oregon State University and University of Nebraska-Lincoln. Report details evaluation findings and recommendations. Report provided to NIFA

Progress 11/01/21 to 10/31/22

Outputs
Target Audience:The primary youth audience of the evaluation is teen attendees of the USDA Secretary's National 4-H Conference. A total of 215 youth delegates attended National 4-H Conference in 2022. During registration, 71% listed Female and 29% listed Male genders. The majority of delegates identified as White (62%), 18% as Black, 6% Hispanic, 5% Asian, 4% Native American, 3% two or more races, and 1% each Pacific Islander and "other." Of the 47 delegations, 37 were from 1862 institutions, 9 were from 1890 institutions, and 1 was from a 1994 institution. Age was normally distributed across ages 14-19 with a mean of 16.7 years. Youth also reported demographic information on the post-survey. A total of 122 youth completed that survey and survey percentages were similar to the registration percentages. Seventy percent of youth identified as white and 30% as another racial or ethnic group. Sixty eight percent were female, 26% male, and 5% as another gender identity. The conference attracted mostly long-term 4-H members, with 79% having five or more years in the program and just 11% having one year or less. Changes/Problems:Actual FTE's for this reporting period are 0.035 (i.e. 3.5% time total for all staff) under Participants. The form rounds to nearest tenth, so box for "nothing to report" must be checked to submit the report. What opportunities for training and professional development has the project provided? Nothing Reported How have the results been disseminated to communities of interest?2022 National 4-H Conference Evaluation Report final report provided to and accepted by USDA NIFA for award 2022-67037-36130. It has not been released publicly by National 4-H Council. Details framework for evaluation, methodology, summative and formative findings, and recommendations. 2022 National 4-H Conference Evaluation Report Summary was provided to and accepted by USDA NIFA, which then distributed the report to the Extension Committee on Organization and Policy (ECOP) for award 2022-67037-36130. It has not been released publicly by National 4-H Council. ECOP serves as the executive board and representative body of the Cooperative Extension System. Details summary evaluation findings and recommendations. What do you plan to do during the next reporting period to accomplish the goals?There were no significant challenges encountered in Year 1 of the award.Action steps in Year 2 will continue successful strategies and expand as needed to address any changes in Conference goals or outcomes. The Year 2 actions steps toward the goals and objectives of this project will include: Meet with the 2023 Conference planning leads to discuss the 2022 report. Share a summary of the year-one findings on the National 4-H Conference information session in December, 2022. Confirm any changes to the goals and outcomes of the Conference. Revise the Theory of Change and evaluation plan as needed. Participate in bi-monthly National 4-H Conference planning team zoom meetings. Revise and develop evaluation tools as needed. Attend the National 4-H Conference in April 2023 and implement the evaluation plan. Analyze the data and develop a 2023 comprehensive report and summary report.

Impacts
What was accomplished under these goals? The project goal is to undertake a comprehensive evaluation, qualitative and quantitative, of the 2022-2025 National 4-H Conference (Conference) events/programs. Evaluation findings and recommendations will be used by USDA NIFA and the Planning Committee to improve delegate experiences overall and ensure each experience on the agenda uniquely supports the overall goals of the Conference. USDA NIFA, LGU 4-H programs and delegates at the next Conference are the most immediate beneficiaries. This was the first comprehensive evaluation of Conference in decades, and included the delegate experience, conference logistics, LGU participation and other partners. The report verified that, overall, Conference is a quality youth development experience in which youth of diverse backgrounds can be successful. Also found were areas where programming could be strengthened to better align with the logic model. The 2022 report will be used as a baseline for subsequent reports. Objective 1: Conduct an annual end-of-event/program survey that assesses the quality of the program for promoting positive youth development and youths' engagement in learning. The Oregon State University Internal Review Board concluded all activities for this grant are Evaluation and not subject to further Human Subjects Research review. A post-survey included items focused on the quality of the delegate experience. Positive Youth Development (PYD) occurs in context, and the chosen quality measures are known to increase PYD outcomes. Items in "Belonging" were related to whether delegates felt physically and socially safe with the other youth and adults at Conference. "Agency" questions focus on whether Conference allowed delegates to experience independence and the mentoring appropriate in a youth-adult partnership. In "Contribution" items delegates reported how Conference helped them to give back to themselves, other delegates, and their home communities. Quality measures are overall positive and encouraging. In Belonging measures, 95% or better of answers were aligned with a positive indication of quality. Only 5% of delegates reported feeling left out all or most of the time. Agency measures responses indicate delegates had some room for independence - 91% did or usually figured things out for themselves, and 95.1% felt that it was always or usually okay to make mistakes. Contribution measures are quite strong; the most positive answer getting more than 75% response in all areas. Even when tightly scheduled, Conference planners should strive for more positive responses about delegate choice and activity preference. Employing more youth-adult partnership best practices (i.e., youth voice in conference planning, more choices during unscheduled time) would likely improve these measures of quality. Objective 2: Conduct an annual end-of-program survey to measure learning outcomes for the program, determined in consolation with NIFA. The youth delegate post-survey was designed to assess the impact of the Conference experience on the outcomes of capacity for leadership, civic engagement and an increased level of agency/empowerment to participate in solving complex issues. More than 90% of delegates intend to seek out civic engagement opportunities, feel a sense of responsibility to solve problems, and feel empowered to do so. The post-survey also included 11 items from the 4-H Common Measures Social Skills Assessment, measuring aspects of personal development and self-awareness that reflect skills needed to make effective group decisions. Mean responses ranged from a low of 3.4 to a high of 3.7 on Likert-type scale of 1 to 4. The highest means are for items related to working with people different from oneself. The lowest are for items reflecting empathy - managing one's own feelings and being thoughtful of others' feelings. One way of interpreting responses is delegates learned to value the views of others but could use more help developing concrete skills to navigate groups with differing views. Objective 3: Conduct potential additional annual youth evaluations as appropriated, such as participatory evaluation with youth and/or follow-up survey(s) after the program concludes. Additional data on youth experience was collected through a daily "rate the day" survey released each evening, providing data about engagement in each session and attitudes about Conference. Monument Tours and Roundtable sessions were the highest rated activities by youth while workshops and service were less engaging. The most used words to describe each day were "fun" and "exciting." A focus group of 18 Collegiate Facilitators was conducted the last day. They had meaningful feedback about their experience and observations of the youth delegates. Suggestions for improvement were incorporated into the final report. A post-survey of LGU contacts was used to evaluate the experience of the LGU in recruitment, registration and participation, and showed great dedication to the program, contributed ideas for registration and logistics, and contained rich anecdotal information about impacts on delegates. A post-survey of federal partners showed that the program was well received by those that completed the survey. Objective 4: Utilize 4-H Common Measures for data collection. 11 items from the 4-H Common Measures Social Skills Assessment were in the post-survey, measuring aspects of personal development and self-awareness that reflect skills needed to make effective group decisions. Findings are under Objective 2. Objective 5: Provide appropriate data analysis. Quantitative data came from registration reports, daily surveys, and the delegate post-survey. Demographics, session ratings, and survey items using Likert-type scales were analyzed with descriptive statistics such as means and rankings. Comparisons of survey responses were made between subgroups using Chi-Square Goodness of Fit Tests. There were no significant differences in youth Belonging, Agency, or Contribution found between racial/ethnic groups, genders, or length of time in 4-H. Retrospective pre/post survey questions were analyzed with a paired T-test. Delegates reported a highly significant increase in their interest in Civic Engagement due to their conference participation. Qualitative data was used to support the quantitative data, for suggestions for improvement and exemplar quotes that demonstrate the impact and importance of the conference. Objective 6: Complete an annual program evaluation report. The 2022 Evaluation Report was completed then submitted to and accepted by NIFA. The report includes all data collected, relevant analysis, and section of conclusions and recommendations. Findings include overall strong measures of youth learning, sense of belonging, and feelings of efficacy. Each program element aligned with the overall program goals, although some elements were more effective and engaging than others. The Roundtable groups, which are the core of the conference, are the strongest element. Data show that delegates gained new knowledge, changed attitudes (particularly around being exposes to new ideas and opinions) and they intend to apply their skills in their local communities. A two-page Evaluation Summary highlighting delegate demographics, notable impacts, and key recommendations was completed, submitted to and accepted by NIFA. NIFA distributed the report to the Extension Committee on Organization and Policy. A stated goal of NIFA is to increase the number of LGUs that send delegations and sharing the summary is expected to aid in recruitment. Objective 7: Provide recommendations for the succeeding year's Conference based on evaluation results. The 2022 Evaluation Report included 40 specific suggestions for change or improvement in the Conference. These included topics such as logistics, LGU communications, planning timeline, facilitator support, and which elements were in need of strengthening.

Publications

• Type: Other Status: Other Year Published: 2022 Citation: Hein, Wendy and Lingard, Jill (2022). 2022 National 4-H Conference Evaluation Report. Oregon State University and University of Nebraska-Lincoln. Details framework for evaluation, methodology, summative and formative findings, and recommendations. Report provided to NIFA for award 2022-67037-36130; not released publicly by National 4-H Council.
• Type: Other Status: Other Year Published: 2022 Citation: Hein, Wendy and Lingard, Jill (2022). 2022 National 4-H Conference Evaluation Report Summary. Oregon State University and University of Nebraska-Lincoln. Details summary evaluation findings and recommendations. Summary Report provided to NIFA which then distributed it to the Extension Committee on Organization and Policy (ECOP) for 2022-67037-36130; not released publicly by National 4-H Council. ECOP serves as the executive board and representative body of the Cooperative Extension System (CES).