NRI-SMALL: FRAIL-BOTS: FRAGILE CROP HARVEST-AIDING MOBILE ROBOTS

• Sponsoring Institution: National Institute of Food and Agriculture
• Project Status: COMPLETE
• Funding Source: AFRI COMPETITIVE GRANT
• Reporting Frequency: Annual
• Accession No.: 1000459
• Grant No.: 2013-67021-21073
• Cumulative Award Amt.: $1,123,463.00
• Proposal No.: 2013-05018
• Project Start Date: Sep 1, 2013
• Project End Date: Aug 31, 2019
• Grant Year: 2013
• Program Code: [A7301]- National Robotics Initiative

Project Director
Vougioukas, S. G.

Recipient Organization
UNIVERSITY OF CALIFORNIA, DAVIS
410 MRAK HALL
DAVIS,CA 95616-8671

Performing Department
Sponsored Programs

Non Technical Summary
Mechanizing the hand harvesting of fresh market fragile crops constitutes one of the biggest challenges to the sustainability of the U.S. fruit and vegetable industry. Depending on the commodity, labor contributes up to 60% of the variable production cost and recent labor shortages have led to significant loss of production. Conventional mechanical and robotic harvesters have not successfully replaced the judgment, dexterity and speed of experienced farmworkers, at a competing cost. As an intermediate to complete mechanization, mechanical labor aids such as conveyor belts and mobile platforms have been introduced to increase worker productivity, by reducing the round-trip walking time to carry the harvested produce to the loading stations. However, the adoption of such labor aids has been very slow. They require large initial investment; field-to-field transport logistics are problematic due to their size, which also restricts their usage in large planar fields; they require skilled human operators, and changes in work practices; and productivity is increased at the cost of ergonomics. The main objective of this project is to lay the scientific and technical foundations for developing teams of inexpensive, relatively small, harvest-aiding mobile robots. These co-bots will support human pickers by supplying them with empty containers and by transporting containers filled with harvested crops to unloading stations. The collective operation of these co-bots is envisioned to offer the services of an alternative, easy-to-deploy fast and robust transport system, which increases productivity, safety and ergonomic metrics using as few robots as possible. Intellectual merit. An innovative approach will be developed, which models the coupled operations of manual harvesting and crop and container transport, as the machine production and materials transport functions of a Flexible Manufacturing System (FMS). Based on this paradigm, stochastic models of the harvesting activities and robot fleet operations will be developed using formalism amenable to numerical optimization. Predicting the spatiotemporal distribution of future transport requests is essential for optimal robot dispatching. Unlike the FMS paradigm where machine production is modeled using probabilistic distributions, in this work the developed stochastic, manual-harvesting model - modulated by crop yield and human work patterns - will be integrated into a model predictive dispatcher that will dynamically match the fleet capacity with pickers' current and predicted transportation demand. Since human operators are involved in all types of agricultural production systems, extensions of this approach could be utilized for optimizing agricultural field logistics of labor-intensive specialty crops as well as highly mechanized commodity crops. Broader impact. The proposed robotic transport system aspires to offer financial benefits for U.S. fruit and vegetable farmers, market advantages for SMEs building advanced agricultural equipment, as well as increased safety for farm workers. Agriculture is often - wrongly - perceived by pupils and students as a 'low tech' field. The truth is that agricultural equipment is getting more 'mechatronic' and farm management information systems are getting more complex. Hence, a new generation of researchers, engineers and professionals will be needed to design, build and operate future complex agricultural production systems. This educational and cultural challenge will be addressed by including K-12 and college students in the research activities of the project, and by incorporating its key findings in graduate curricula. Finally, the exposure of growers and unskilled, low-income farm laborers to robotic technology through the planned experiments and dissemination activities could help increase their openness to technology, and to the education and training required to utilize it.

Animal Health Component

20%

Research Effort Categories

Basic

0%

Applied

20%

Developmental

80%

Classification

Knowledge Area (KA)	Subject of Investigation (SOI)	Field of Science (FOS)	Percent
402	1122	2020	100%

Knowledge Area
402 - Engineering Systems and Equipment;

Subject Of Investigation
1122 - Strawberry;

Field Of Science
2020 - Engineering;

Keywords

efficiency

ergonomics

fruit

labor

robotic

transport

Goals / Objectives
The main objective of this proposal is to lay the scientific and technical foundations for developing teams of inexpensive, relatively small, harvest-aiding mobile robots (FRAIL-bots), which will support human pickers by supplying them with empty containers and by transporting containers filled with harvested crops to loading stations at the edges of a field. The collective operation of these 'robotic carts' is envisioned to offer the services of a fast and robust crops transport system that can be easily deployed in the field and can serve requests for point-to-point transport in real time. In order to achieve this, two major goals have been identified: The first goal s to develop stochastic models of the coupled manual crop picking and transport operations, which are expressed in a suitable formalism that can be used for calculating robot dispatching and routing. We propose to accomplish this by developing a mixed logical dynamical (MLD) description of the coupled harvest-transport system, which is amenable to numerical optimization and control. The second goal is to integrate the stochastic harvesting and robot-based crops transport models into a predictive dispatcher/router, which will incorporate the stochastic spatiotemporal predictions of future transport requests to achieve improved performance. We propose to accomplish this by incorporating Monte Carlo integration of the stochastic MLD model into a one-step predictive controller.

Project Methods
Manual harvest and transport activities modeling The geometric layout of crops planted in rows will be exploited to represent the workspace as a graph. Field row traversal by workers and robots will be represented as graph traversal. Harvest-related activities along field rows involve simple motions and mass transfer between involved agents. These dynamics depend on worker and robot operating states/modes; hence the coupled and coordinated activities will be expressed as interacting dynamic systems. Workers may switch between operating modes either driven by external events, or at-will (e.g., picking and resting). Hence, stochastic transitions exist in the manual-picking model, rendering the coupled system as a stochastic hybrid system. This coupled system will be expressed as a mixed logical dynamical (MLD) system, i.e., a system described by dynamic equations and constraints involving continuous and binary/logic variables. The reason is that FRAIL-bots will be dispatched and routed in a closed-loop fashion using mixed integer optimization, and MLD models have been successfully used for this purpose. The challenges undertaken include the common representation of harvesting and robot operations as hybrid systems and equivalent MLD models. Well-established correspondences between state machines, propositional calculus and linear integer programming will be used. Dynamic scheduling and routing of robot fleet Information about the spatiotemporal distribution of future requests has been shown to improve the performance of dynamic pickup and delivery systems. Therefore, a one-step-ahead model predictive control approach will be pursued for dispatching and routing FRAIL-bots. The controller will integrate forward in time the coupled MLD models of manual harvesting and robot transportation, to predict the next transport requests from all workers; these will be used to re-compute robot dispatching and routing. The challenge here is that, due to the stochastic nature of the manual picking model, integration using a certain instance of model parameters -e.g., picking rate - would generate only a non representative sample of the system's stochastic trajectories. We propose to use a particle-based Monte Carlo integration approach that estimates the spatiotemporal distribution of future requests by adequately sampling the stochastic and time varying parameters of the manual-harvesting model. The choice of objective function and constraints pose open problems that will be addressed. The total time that the freshly harvested crop remains in the field before refrigeration must be minimized because the percentage of marketable fruit drops dramatically as a function of this time. Also, the maximum waiting time of individual workers must be minimized. Increasing overall performance by delaying a worker's job cycle - incurring less pay - would certainly cause the workers to reject the system. Additionally, ergonomic metrics must also be incorporated - either in the cost function or as soft constraints - so that productivity is increased in-step with safety and ergonomic metrics, as mandated by federal and state regulations.

Progress 09/01/13 to 08/31/19

Outputs
Target Audience:One post-doctoral researcher, two Ph.D. students (one woman), one engineering MSc student and one undergraduate student were mentored in the context of the project; informal laboratory instruction was the key instrument. Our research team worked closely with growers in Santa Maria, CA and conducted field experiments and time studies in their fields. One Ph.D. student presented findings at the 2019 ASABE Intl. Meeting in Boston, MA to researchers, students, and academics. The PI gave several presentations related to robotic harvest-aids. The audiences were growers, entrepreneurs, researchers, students, academic staff and the general public. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?One post-doctoral researcher, two Ph.D. students (one woman), one engineering MSc student and one undergraduate student were mentored in the context of the project; informal laboratory instruction was the key instrument. Our research team worked closely with growers in Santa Maria, CA and conducted field experiments and time studies in their fields. One Ph.D. student presented findings at the 2019 ASABE Intl. Meeting in Boston, MA to researchers, students, and academics. How have the results been disseminated to communities of interest?Results of this research were presented at the 2019 ASABE Intl. Meeting in Boston, MA. Also, the PI gave several presentations related to the project and robotic harvest-aids, in general. The audiences were growers, entrepreneurs, researchers, students, academic staff and general public. December 3, 2018. Almond Board of California, Davis, CA. January 30, 2019. Cling Peach Board, Sacramento, CA. February 5, 2019. California Pear Board, Davis, CA. February 13, 2019. Morning Star Company, Davis, CA. March 19, 2019. World Bank Headquarters, Washington, D.C. April 1, 2019. Western Center for Agricultural Health and Safety, Davis, CA. April 3, 2019. California Strawberry Commission, Cal Poly at Saint Luis Obispo. August 27, 2019. Taylor Farms, Salinas, CA. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? ENTIRE PROJECT IMPACT The project demonstrated that low-cost robots could be used as harvest-aids in strawberry harvesting, to transport trays to and from the pickers. Robots can reduce pickers' nonproductive walking to deliver trays and increase efficiency by 10% to 21%, depending on the picker to robot ratio, robot scheduling method, yield, and crew performance. Commercialization of this technology could provide some relief to growers impacted by labor shortages. The technology can be adapted to other crops, like, for example, table grapes. Also, an instrumented strawberry picking cart was developed that can be used to generate strawberry yield maps with 5% accuracy. This new capability - if commercialized - can enable precision management of strawberry fields and crops. Objective 1. Develop stochastic models of picking and transport operations 1) Major activities completed / experiments conducted. A mobile sensing system was built, and software was developed to collect worker hand motion data during strawberry picking. A picking cart was built to measure cart location, and the weight of the harvested strawberries in real-time. The data are transmitted from the cart to the robots wirelessly and used for predictive robot scheduling. The cart could be used to generate accurate strawberry yield maps at high resolution. A new modeling framework was developed for human-robot collaboration during harvesting, using hybrid systems formalism that combines discrete and continuous dynamics (picker-robot motions and crop/mass transfer). A simulator was calibrated using data from field experiments. 2) Data collected. Harvest-related data were collected during commercial harvesting in all years of the project, from videos and manual observation (e.g., picking rate; timestamps of activities; location of each picker; walking speeds, and distances covered). 3) Summary statistics and discussion of results. The data collected from harvesting operations were processed, and picker-related performance parameters were estimated. The time it takes a picker to harvest a full tray was 300s and could be predicted with an accuracy of 9%. The average walking speed for tray transportation was 1.4 m/s, and the average time waiting in a line (with a tray) to deliver fruit was 19 s. 4) Key outcomes or other accomplishments realized. A harvesting simulator was developed and calibrated using the data from the experiments. The simulator predicted the pickers' non-productive time with a maximum error of 6.4%. An instrumented strawberry picking cart was developed that can be used to generate strawberry yield maps with 5% accuracy. This new capability - if commercialized - enables precision management of strawberry fields and crops. Objective 2. Integrate models and predictive dispatcher 1) Major activities completed / experiments conducted; The reactive version of the transport scheduling problem was modeled, integrated with the harvesting simulator, and solved. In this version, robots start their trips to pickers when a transport request is initiated (no prediction). Policies like First-Come-First-Serve and Longest/Shorted-Distance-First were implemented. The problem was solved for robot teams of different sizes. A predictive scheduling algorithm was developed and integrated with the harvesting simulator and was used to analyze the robot team performance in the presence of uncertainty. Several small-footprint robot versions were built during the duration of the project. Finally, two larger robots -that straddle a strawberry bed- were developed, along with their software, which implements autonomous robot navigation; communications between picking carts, robots and field computer; reactive and predictive scheduling. An economic model was developed that characterizes optimal adoption of the robot under alternative parameters, including acquisition and annual maintenance costs, operation cost per hour, people-to-robot ratio, and time savings. The model was implemented as Excel spreadsheets with embedded equations and was used to generate adoption scenarios. 2) Data collected. Data were generated in the summer of 2019, by using the developed simulator, using reactive and predictive scheduling policies for the robot team. Different picker-robot ratios were tested. Robot navigation experiments were performed in Salinas, CA, in summer 2019, and data were collected to assess robot localization and path tracking accuracies. 3) Summary statistics and discussion of results. Experiments with a 25-picker crew showed that deploying five crop-transport robots increased the harvest efficiency by 10.2% when reactive scheduling was used. With more than five robots, the scheduling policies performed similarly, i.e., the efficiency did not increase more than 93.2%. Simulations using predictive scheduling showed that when five robots were used, the efficiency gain was 15.3%, going up from 77% for all-manual harvesting, to 92.3% for robot-aided harvesting. When ten robots were used, the efficiency reached 98.4%, an increase of 21.4%. The experimental results showed that GNSS-based path tracking accuracy ranged from 2 to 8 cm, which was adequate to enter and exit strawberry furrows - and travel inside them - without any problems. The robots would always enter the correct furrow, where the picking cart was located, and stop at a pre-specified distance before reaching the cart. 4) Key outcomes or other accomplishments realized. Low-cost ($10k), robots can be used as harvest-aids in strawberry harvesting, to transport trays to and from the pickers. Robot navigation can rely on GNSS. Predictive scheduling is necessary for the robots to achieve large efficiency gains (15% - 21%, for 5 and 10 robots, respectively, and 25 pickers). FINAL YEAR A novel approach was finalized and implemented in a stochastic simulator, to model the collaboration of human pickers and tray-transporting robots, in the context of strawberry harvesting. The approach uses hybrid automata, a formalization that allows simulation, and can be used as an executable model of the harvesting operation that can be used for real-time robot control and optimization of robot scheduling. Data collected from pickers harvesting in commercial strawberry fields in CA, during the summer of 2019, were used to calibrate the simulator, which predicted the pickers' non-productive time with an error that ranged between 1.2% - 6.4. Harvesting models were finalized, and a stochastic prediction model was developed for tray transport requests, which had an accuracy of 3% - 10%. The model relies on real-time picking rate data provided by an instrumented picking cart. The cart design was finalized in the last year. Robot-aided simulation results for a 25-picker crew and three reactive scheduling policies showed that when deploying 3, 4, or 5 robots, serving the closest picker first, outperformed the other two scheduling policies (prioritize the most distant picker, or the first picker to request a robot); however, serving the earliest request first, resulted in lower variance (more consistent service). Deploying five crop-transport robots increased the harvest efficiency by 10.2%. With more than five robots, the scheduling policies performed similarly. Simulations using predictive scheduling showed that when five robots were used, the efficiency gain was 15.3%. However, when 10 robots were used, the efficiency reached 98.4%, an increase of 21.4%. Two fully functional robots were developed. Their software implements autonomous robot navigation; communications between picking carts, robots and field computer; reactive and predictive scheduling. Robot navigation experiments were performed in Salinas, CA, in summer 2019, and data were collected to assess robot localization and path tracking accuracies. Also, instrumented strawberry picking carts operated inside furrows and their signals were transmitted to the field computer, which dispatched the robots.

Publications

• Type: Journal Articles Status: Published Year Published: 2018 Citation: Khosro Anjom, F., Vougioukas, S. G., Slaughter, D.C. (2018). Development and Application of a Strawberry Yield-Monitoring Picking Cart. Computers and Electronics in Agriculture. 155: 400-411.
• Type: Journal Articles Status: Published Year Published: 2019 Citation: Khosro Anjom, F., Vougioukas, S. G. (2019) Online Prediction of Tray-Transport Request Time for Robot-Aided Strawberry Harvesting using Mechanistic Grey Models. Biosystems Engineering. 188:265-287.
• Type: Journal Articles Status: Submitted Year Published: 2019 Citation: Seyyedhasani, H., Peng, C., Jang, W., Vougioukas, S.G. (2019). Collaboration of Human Pickers and Crop-transporting Robots during Harvesting - Part I: Model and Simulator Development. SUBMITTED. Computers and Electronics in Agriculture.
• Type: Journal Articles Status: Submitted Year Published: 2019 Citation: Seyyedhasani, H., Peng, C., Jang, W., Vougioukas, S.G. (2019). Collaboration of Human Pickers and Crop-transporting Robots during Harvesting - Part II: Simulator Evaluation and Robot-Scheduling Case-study. SUBMITTED. Computers and Electronics in Agriculture.
• Type: Conference Papers and Presentations Status: Published Year Published: 2019 Citation: Peng, C., Vougioukas, S.G. (2019). Scheduling performance of harvest-aiding crop-transport robots under varying earliness in access to transport-request predictions. ASABE Annual International Meeting. Boston, Massachusetts.

Progress 09/01/17 to 08/31/18

Outputs
Target Audience:One undergraduate female student was offered internships in my Lab and gathered practicum experience in strawberry fields in Santa Maria, CA for manual and machine-aided harvesting. The PI and co-PIs mentored two post-doctoral researchers, two Ph.D. students (one woman), and one engineering MSc student in the context of the project; informal laboratory instruction was the key instrument. Our research team worked closely with growers in Santa Maria, CA and conducted field experiments and time studies in their fields. One postdoctoral researcher and one graduate student presented findings at the 2018 ASABE Intl. Meeting at Detroit, MI to researchers, students and academics. The PI gave several presentations related to robotic harvest-aids. The audiences were growers, enterpreneurs, researchers, students, academic staff and general public. Changes/Problems:Extensive testing in summer 2018 in challenging strawberry fields in Santa Maria, CA, showed that these robots' stability and ability to robustly traverse furrows in fields is not adequate for the large scale experiments we need to perform (e.g., 1-3 robots for 10 pickers). The ground surface of furrows can be extremely rough for smaller robots to traverse; it becomes so, when it is traversed by heavy machinery after excessive irrigation. Therefore, we applied for and received a one-year no cost extension for the project. What opportunities for training and professional development has the project provided?One undergraduate female engineering student was mentored in the context of the project. She participated in field experiments and worked on electronics and calibration of the strawberry picking cart. Two post-doctoral researchers and three engineering graduate students (one female) were also mentored and trained in the context of the project. One graduate student and one postdoctoral researcher presented findings at the 2018 ASABE Intl. Meeting. How have the results been disseminated to communities of interest?Results of this research were presented at the 2018 ASABE Intl. Meeting in Detroit, MI. Also, the PI gave several presentations related to robotic harvest-aids. The audiences were growers, enterpreneurs, researchers, students, academic staff and general public. 2017 October, 15. From Agriculture to Activism: Digital media across Fields. UC Davis. 2017 October, 2. Agricultiral Robotics. UC Davis. Gowers delegation from Argentina. 2018 January, 8, 18. Mechanized harvsting. Modesto, CA. Cherry and clig peach growers. 2018 January, 11. Strawberry Automation Summit. California Strawberry Commission, Cal Poly at Saint Luis Obispo. 2018 March, 2. "Bio Automation Lab R&D Activities". 2018 April, 23. "Labor Saving Technologies". Innovator Summit, UC Davis. 2018 August, 22. "Bio Automation Lab R&D Activities". Trimble - UC Davis. What do you plan to do during the next reporting period to accomplish the goals?We already built a new prototype that is larger and straddles the strawberry beds to ensure stability even in adverse field terrain conditions. We will finalize this prototype, test it thoroughly, build two more copies and perform the large-scale experiments needed to evaluate harvest-aids in real (commercial) harvesting strawberry conditions.

Impacts
What was accomplished under these goals? A. First goal: Develop stochastic models of picking and transport operations Data from the cart and from all field experiments were used to develop stochastic models of picking and tray transport request activities. The cart made it possible to collect strawberry yield data in an automated fashion, foe the first time ever. Results from this work are included in a Ph.D. dissertation (Khosro Anjom, 2018) and peer-reviewed published or submitted articles listed in this report. A detailed simulator was developed that models all crew harvest activities and robot team activities. Essentially, a new modeling framework was developed for human-robot collaboration during harvesting, using a hybrid systems formalism that combines discrete dynamics (Finite State Machines) with continuous dynamics (picker-robot motions and crop/mass transfer). The simulator was calibrated using data from our field experiments. Results from this work are included in a M.Sc. thesis (Jang, 2018) and a submitted manuscript. An economic model was developed that characterizes optimal adoption of the robot under alternative parameters, including acquisition and annual maintenance costs, operation cost per hour, people-to-robot ratio, and time savings. The model is implemented as Excel spreadsheets with embedded equations and is currently being used to generate adoption scenarios. A manuscript with labor saving analyses is under preparation. B. Second goal: Integrate models and predictive dispatcher 1. The reactive version of the transport scheduling problem was modeled, integrated with the harvesting simulator and solved. In this version, robots start their trips to pickers when a transport request is initiated (no prediction). Policies like First-Come-First-Serve and Longest/Shorted-Distance-First were implemented. The problem was solved for robot teams of different sizes to investigate the dependence of waiting times on team size. Results revealed that waiting times decrease as a power law of the number of robots and that predictive scheduling is necessary to reduce/eliminate waiting times due to the distance that robots must travel to reach pickers. Results of this work were included in a M.Sc. thesis (Jang et al., 2018) and conference papers (Peng et al., 2018; Seyyedhasani et a;., 2018). 2. A predictive scheduling/dispatching algorithm was developed - and is being fine-tuned - that uses the Multiple Scenario Approach (MSA). A large number of picking request scenarios is generated by sampling the stochastic predictive models developed in this project in the context of the first goal and an optimal, robust schedule for the robot team is generated. The algorithm has been integrated with the harvesting simulator and is currently used to analyze the robot team performance in the presence of uncertainty, as a function of various picker-to-robot ratios. A relevant manuscript is under preparation for submission. 3. Several robot versions had been designed, built, programmed and tested during the duration of the project (Jang, 2018;). The two major versions were small-footprint robots (12 inches wide) that could fit inside a single furrow, whereas a third version had active balance control (Fig. 3c). Extensive testing in summer 2018 in challenging strawberry fields in Santa Maria, CA, showed that these robots' stability and ability to robustly traverse furrows in fields is not adequate for the large scale experiments we need to perform (e.g., 1-3 robots for 10 pickers). The ground surface of furrows can be extremely rough for smaller robots to traverse; it becomes so, when it is traversed by heavy machinery after excessive irrigation.

Publications

• Type: Journal Articles Status: Published Year Published: 2018 Citation: Khosro Anjom, F., Vougioukas, S. G., Slaughter, D.C. (2018). Development and Application of a Strawberry Yield-Monitoring Picking Cart. Computers and Electronics in Agriculture.155:400-411.
• Type: Theses/Dissertations Status: Published Year Published: 2018 Citation: Khosro Anjom, F. (2018). Predictive Modeling of the Temporal Distribution of Tray-Transport Requests for Robot-Aided Strawberry Harvesting. Ph.D. Dissertation. University of California, Davis.
• Type: Journal Articles Status: Under Review Year Published: 2018 Citation: Khosro Anjom, F., Vougioukas, S. G. (2018) Online Prediction of Tray-Transport Request Time for Robot-Aided Strawberry Harvesting using Mechanistic Grey Models. Biosystems Engineering.
• Type: Theses/Dissertations Status: Published Year Published: 2018 Citation: Jang, W.J., (2018). Investigation on the Harvest-aid Robot Scheduling Problem and the Implementation of Its Simulation Platform. M.Sc. Thesis. University of California, Davis.
• Type: Conference Papers and Presentations Status: Published Year Published: 2018 Citation: Peng, C., Seyyedhasani, H., Vougioukas, S.G., (2018). Optimized predictive dispatching of robotic harvest- aids using Multiple Scenario Approach. ASABE Annual International Meeting. Paper Number # 1801694, Detroit, Michigan.
• Type: Journal Articles Status: Published Year Published: 2018 Citation: Khosro Anjom, F., Vougioukas, S. G., Slaughter, D.C. (2018). Development of a Linear Mixed Model to Predict the Picking Time in Strawberry Harvesting Processes. Biosystems Engineering. (166): 76-89.
• Type: Conference Papers and Presentations Status: Published Year Published: 2018 Citation: Seyyedhasani, H., Peng, C., Vougioukas, S.G., (2018). Efficient Dispatching of a Team of Harvest-aid Ro- bots to Reduce Waiting Time for Human Pickers. ASABE Annual International Meeting. Paper Number # 1801715, Detroit, Michigan.

Progress 09/01/16 to 08/31/17

Outputs
Target Audience:Two undergraduate students were offered internships in my Lab and gathered practicum experience. One student (female) performed time studies in strawberry fields in Oxnard and and Saint Luis Obispo for manual and machine-aided harvesting. The other student (male) worked on robot testing and algorithm development. This experience helped him secure a job upon graduation with Trimble. The PI and co-PIs mentored one post-doctoral researcher, three engineering PhD (one woman), tand wo engineering MSc students in the context of the project; informal laboratory instruction was the key instrument. Our research team worked closely with growers in Watsonville, Oxnar and Saint Luis Obispo, CA and conducted field experiments and time studies in their fields. The PI and a graduate student presented findings at the 2017 ASABE Intl. Meeting at Spokane, Washington to researchers, students and academics. The PI gave several presentations related to robotic harvest-aids. The audiences were growers, packers, processors, enterpreneurs, researchers, students, academic staff and general public. Changes/Problems:Hardware and software integration was more complicatd than expected and a no-cost extension was approved. Integration will be completed by late spring 2018, for the robot team and for the picking carts. An additional post-doctoral researcher was recruited to help with this process. What opportunities for training and professional development has the project provided?Two undergraduate students were mentored in the context of the project. One student (female) performed time studies in strawberry fields for manual and machine-aided harvesting. The other student (male) worked on robot testing and algorithm development. This experience helped him secure a job upon graduation with Trimble. One post-doctoral researcher, three engineering PhD (one woman), and wo engineering MSc students were mentored and trained in the context of the project. One graduate student presented findings at the 2017 ASABE Intl. Meeting at Spokane, Washington. How have the results been disseminated to communities of interest?Results of this research were presented at the 2016 ASABE Intl. Meeting in Spokane. The PI gave several presentations related to robotic harvest-aids. The audiences were growers, packers, processors, enterpreneurs, researchers, students, academic staff and general public. 2016 December 6. Harvest mechanization technology, UC Davis ARC; 2016 Joint Vegetable Crops Program Team and Weed Workgroup Meeting. 2017 March 8, Robotics in Ag Tech, UC Merced; 2017 CITRIS Agricultural Technology Fair. INVITED SPEAKER in Panel for Robotics in Ag Tech. 2017 April 4, Addressing Labor Shortages in Agriculture with Robotics & Automation, UC Davis Conference Center. Silicon valley Forum. 2017 April 6, Addressing Labor Shortages in Agriculture with Robotics & Automation, UCD 1207 Robert Mondavi Institute, South Building; BFTV Cluster presentation. 2017 April 27, Robotics in Agriculture, World Food Center Precision Ag Seminar Series. What do you plan to do during the next reporting period to accomplish the goals?Hardware and software integration will be completed by late spring 2018. Robot-aided harvesting experiments will be performed in the of summer 2018.

Impacts
What was accomplished under these goals? A. First goal: Develop stochastic models of picking and transport operations 1. Major activities and experiments 1a. Static and dynamic calibration procedures were developed for the instrumented strawberry picking cart which is used to measure the weight of the harvested strawberries in real time, as the picker fills up the tray with strawberries. 1b. Algorithms were developed to generate yield maps from the data recorded from the picking cart. 1c. Bayesian updating of the stochastic model of tray fill-up time was completed and tested using recorded data from field experiments. 1d. Algorithms were developed for real-time prediction of the expected time - and associated standard deviation - that a picker will need to fill up a tray with strawberries. The time prediction is updated continuously; this is important for proactive robot dispatching. Assessment of the accuracy of the algorithms iwas initiated and is still being conducted. 1e. Software was developed for Kalman-based localization of the robot in the field and for path tracking control using the pure-pursuit algorithm. 2. Data collected 2a. Data were collected during commercial harvesting from July 2017 until September 2017. Timing data was recorded pertaining to tray fill-up times and picking rates for both manual and machine aided strawberry harvesting. 2b. Motion data were collected from the v2. robot during navigation outside of furrows and were used to evaluate the performance of the path-tracking algorithm. 3. Discussion of results The performance of the picking cart was very satisfactory. The calibration procedures were successful and the error in measured strawberry weight was less than 2%. Also, yield maps were successfully generated after harvesting was done. The algorithm for real-time prediction of tray fill-up time was completed and preliminary results are positive. The performance of the v2 robot was still not entirely satisfactory. Its stability inside furrows was not robust and the skid steering system would often slip, thus introducing path-tracking errors. A prototype with an inverted pendulum mechanism for the tray was developed, but its dynamic response could not be fast enough, given the weight of a full strawberry tray and the gimbal system, and the size of motors and batteries. For these reasons, a new/final robot version is being developed that has larger wheels, lower center of gravity, and combines differential drive with independently steered front and rear wheels. 4. Key accomplishments Static and dynamic calibration of the picking cart was completed and development of algorithms for yield mapping were completed and successfully tested. Mechanistic modeling of tray fill-up time was completed and successfully tested. Robot localization and path-tracking algorithms were developed and tested. B. Second goal: Integrate models and predictive dispatcher 1. Major activities and experiments 1a. The Monte-Carlo simulator that incorporates the tray fill-up stochastic model, picker moving speeds and non-productive times and robot dispatching algorithms was extended to include delays due to robot motion when the robot is guided by a path tracking controller. This addition provides more realistic simulations of the coupled human-robot system. 1b. The implementation of a Multiple Scenario Analysis algorithm was initiated and is still under development. In this algorithm the stochastic variables relating to picking are sampled and multiple simulation instances are executed to calculate the corresponding optimal dispatching policies. From the pool of policies, the one best matching current requests is selected. Alternatively, existing schedules are perturbed to match current requests. 1c. An economic model was developed to explore the cost-effectiveness of robotic harvest-aids. The model is currently being populated with parameter values (e.g., robot cost, time savings, labor cost, etc.) so that scenarios can be explored. 2. Data collected Field data collection was performed during commercial harvesting from June 2017 until October 2017. Recorded data included picker walking speeds during picking and during fruit transport to the collection station and return (non-productive times), during manual harvesting. Also, data were generated by running the simulator with various picking crew and robot fleet sizes. These data provide statistics of the picker waiting times (for a robot to arrive) and the overall system efficiency. 3. Discussion of results Harvesting activities for manual and robot-assisted harvesting were simulated, including parameters such as field shape and dimensions, crew size, robot fleet size, traveling velocities, and handling times. Dynamic scheduling policies such as "Shortest Processing Time" and "First Come First Served" were implemented and performance indicators like picker waiting time, robot utilization rate, and robot traveling distances/time were calculated. Simulation results showed robot fleet size is the primary factor affecting picker waiting time. In a configuration where robot team size increased from 3 to 6, the mean waiting time dropped 9%, and the maximum waiting time dropped 57%. Differences among scheduling policies diminish when robot-picker ratio increases, and all policies generate the same schedule when the ratio is high enough. 4. Key accomplishments Execution of harvesting scenarios using the Monte-Carlo harvesting simulator. Development of economic model for robotic strawberry harvest-assist operations.

Publications

• Type: Journal Articles Status: Accepted Year Published: 2017 Citation: Anjom, F.K., Vougioukas, S. G., Slaughter, D.C. (2017). Development of a Linear Mixed Model to Predict the Picking Time in Strawberry Harvesting Processes. Biosystems Engineering.
• Type: Conference Papers and Presentations Status: Published Year Published: 2017 Citation: Wei-jiunn, J., Vougioukas, S.G. (2017). Evaluation of dynamic robot dispatching strategies using a strawberry-harvesting simulator. ASABE Annual International Meeting. Paper Number 1701402. Spokane, Washington.

Progress 09/01/15 to 08/31/16

Outputs
Target Audience:Two undergraduate engineering students (one woman) were offered internships in my Lab and gathered practicum experience on robot testing and algorithm development. The PI and co-PIs mentored two engineering PhD (one woman) and three engineering MSc students in the context of the project; informal laboratory instruction was the key instrument. A graduate-level course on Agricultural Robotics was taught for the second time in Winter 2016 (EBS289). Ten graduate students participated in formal classroom instruction and lab work in a hybrid learning environment. Our research team worked closely with one large grower in Watsonville, CA and conducted field experiments in his fields. The PI mentored a team of students from the Davis Senior High School for a summer project that developed a gripper for fruit picking. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?Two PhD students were mentored in the context of the project. Also, internships were provided for three graduate (1 PhD, 2 MSc) and two undergraduate students, and one high-school student. A team of three seniors deigned, fabricated and tested the snap-on yied monitor device as their senior design project. One PhD student participated in and presented a poster at the 2016 ASABE International Meeting in Florida. How have the results been disseminated to communities of interest?The results of this reporting period were presented to the technical and scientific communities at the 2016 ASABE International meeting in Florida. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? A. First goal: Develop stochastic models of picking and transport operations 1. Major activities and experiments 1a. A novel device was built that can be snapped on to a standard picking cart, and can measure the weight of the harvested strawberries in real-time, as the picker fills up the tray with strawberries. The device has only two - instead of four - load cells and contains an off-the-shelf RTK GPS module to collect precise location data, in order to produce yield maps. The data is stored on a SD card. 1b. A new version (v2) of the strawberry robot was designed and built. This version has four independently actuated wheels and a suspension system that helps it navigate over rough terrain. The robot has an off-the-shelf RTK GPS module, and odometry and inertial navigation sensors. 1c. Stochastic modeling of tray fill-up time was completed. This model is a mixe linear model and can take into account factors such as time-of-day, plant spacing, and picker speed (fast/slow) in order to predict a mean value and a standard deviation for the time rewuired to fill-up an empty tray during strawberry harvesting. 2. Data collected 2a. Field data collection was performed during commercial harvesting from June 2016 until October 2016. During the experiments timing data was recorded relating to tray fill-up times, and picking rates (kg/min) for both manual and machine-aided strawberry harvesting. 2b. Data was collected from the v2. robot as it traversed furrows in commecial fields. Data inlcuded position, Euler angles, velocities, and accelerations. 3. Discussion of results The stochastic model for the tray fill-up time is statistical; hence it will always contain some error, depending on the individual picker. Therefore, online updating of the model based on true fill-up times will be required. The v2 robot would become unstable (roll over) in cases when the ground of a furrow was extremely rough. For this reason, a v3 is being developed that actively controls its center of gravity by rotating its chassis like an inverted pendulum. 4. Key accomplishments Design, fabrication and testing of a picking cart snap-on device for yield monitoring. Design, fabrication and testing of an improved robot cart. Completion of stochastic model for tray fiil-up time. B. Second goal: Integrate models and predictive dispatcher 1. Major activities and experiments A Monte-Carlo simulator was developed in Matlab that incorporates tray fill-up stochastic model as well as picker moving speeds and non productive times with robot dispatching algorithms. 2. Data collected Field data collection was performed during commercial harvesting from June 2016 until October 2016. During the experiments data was recorded relating to pickers walking speeds and non productive times, during manual harvesting. Also, data was generated by running the simulator with various picking crew and robot fleet sizes. These data provide statistics of the picker waiting times (for a robot to arrive) and the overall system efficiency. 3. Discussion of results Resuts showed that the ratio of robots to humans must excedd 1:3, when reactive dispatcing is used, such as first-come-first-serve. However, much larger ratios cannot improve performance because there is a minimum waiting time for any given transport requrest, which is dictated by the distance the robot needs to travel to get to a picker. Therefore, tray fill-up time prediction is essential and is being incorporated currently via a Multiple Scenario Analysis approach. This approach samples the stochastic variables relating to picking (e.g., tray fill-up time, picker moving speed) and runs multiple simulation runs and calculates the corresponding optimal dispatching policies. From the pool of policies, it selects the one best mathing current requests. 4. Key accomplishments Completion of Monte-Carlo harvesting simulator.

Publications

• Type: Other Status: Published Year Published: 2016 Citation: Khosro Anjom F., Vougioukas, S. (2016) Prediction of Picking Time in Strawberry Harvesting Using A Conditional Linear Mixed Model. Poster presentation. ASABE Intl. Meeting, Florida, USA.
• Type: Journal Articles Status: Submitted Year Published: 2016 Citation: Khosro Anjom F., Vougioukas, S., Slaughter, D. 2016. Stochastic Modeling of Tray Fill-Up Time in Strawberry Harvesting. Submitted to Biosystems Engineering.

Progress 09/01/14 to 08/31/15

Outputs
Target Audience:Two undergraduate and two graduate (MSc candidate) students were offered summer internships in my Lab and worked on robot testing and algorithm development. The PI and co-PIs mentored two PhD and two MSc students in the context of the project; informal laboratory instruction was the key instrument. A new graduate-level course on Agricultural Robotics was developed and taught in Winter 2015 (EBS289). Ten graduate students from four Departments enrolled and finished the course. Our research team worked closely with one large grower in Watsonville, CA and conducted field experiments in his fields. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?Two PhD students and two MSc students were mentored in the context of the project. Also, summer internships were provided for graduate and undergraduate students. The PI and one PhD student participated in and presented at the 2015 ASABE International Meeting in Montreal. How have the results been disseminated to communities of interest?The results of this reporting period were presented to the technical and scientific communities at the 2015 ASABE International meeting in New Orleans. Also, results from this work were presented on March 18th during a "Research Exchange" webcast organized by CITRIS@Berkeley and transmitted to 4 UC campuses and uploaded on YouTube. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? A novel strawberry picking cart was developed that can be used to produce yield maps for strawberry fields. Yield maps show spatial variability in crop production and constitute an invaluable tool for precise crop management. This cart also records workers' picking rates, thus making it possible for automated harvest-aid machines to monitor the productivity and ergonomics of strawberry pickers while they are picking. Using this information, automated harvest-aids will be able to increase picker efficiency without compromising their safety A. First goal: Develop stochastic models of picking and transport operations 1. Major activities and experiments. 1a. A prototype picking cart was built and instrumented to measure the weight of the harvested strawberries in real-time, as the picker fills up the tray with strawberries. The cart has a novel off-the-shelf RTK GPS module to collect precise location data, in order to produce yield maps. 1b. A holding base for the strawberry carton-tray was designed, built and tested. The base is part of the robot and secures strawberry clams while isolating them from vibration during travel in order to avoid mechanical damage. 1c. A miniature sensor was developed to monitor picker hand motions. The sensor is inserted in a pocket of a picking glove and transmits data to a device worn on the picker's belt. 2. Data collected. Field data collection was performed during commercial harvesting from June 2015 until October 2015. During the experiments timing data for pickers were recorded, and data related to the functionality of systems 1a, 1b and 1c were collected. Such data include harvested strawberry mass per unit time for each picker, timestamps of each component of the picking process of individual pickers, location of each picker as a function of time, and vibration signals for the robot as it carries strawberries in the field. 3. Discussion of results The collected data confirmed that the developed robot holding-base and its isolating system for the strawberry tray (system 1b) kept vibration below the threshold that can induce mechanical damage to the transported strawberries. Also, a yield map was generated for an approximately 0.12 ha area plot of the field using the cart (system 1a). Load cell measurements showed a mean absolute percentage error of 1.6% compared to the weights recorded by a digital scale. Finally, the hand motion sensor successfully measured and transmitted data at a rate of 100 Hz, which is more than adequate to describe hand motion during strawberry picking. 4. Key accomplishments Successful employment of picking cart that recorded picking rates and yield map. Completion of strawberry tray carrying system on the robot that isolates strawberries from potentially harmful ground-induced vibration. Completion of hand motion measuring system. B. Second goal: Integrate models and predictive dispatcher 1. Major activities and experiments. The static crop transport scheduling and routing problem was enriched with non-collision constraints and was solved optimally using the CPLEX MILP solver. The problem was solved for robot teams of different sizes to investigate the dependence of waiting times on team size. 2. Data collected. No data collection was necessary for this activity. Data were created by the simulations. 3. Discussion of results The maximum waiting time of pickers for a robot to arrive decreases exponentially as the size of the robot team increases. This is similar to the case where no collision constraints exist; however, the waiting times can be larger. The dynamic crop transport problem will be modeled in the next project report period. 4. Key accomplishments The optimization of crop transport scheduling and routing problem with collision constraints.

Publications

• Type: Journal Articles Status: Under Review Year Published: 2016 Citation: Vougioukas, S.G., He, L., Arikapudi, R. (2016). Orchard Worker Localisation Relative to a Vehicle Using Radio Ranging and Trilateration. Biosystems Engineering.
• Type: Conference Papers and Presentations Status: Accepted Year Published: 2015 Citation: Farangis Khosro, A., Rehal, R., Vougioukas, S. (2015). A Low-Cost, Efficient Strawberry Yield Monitoring System. ASABE Annual Intl. Meeting; Paper Number 152189408, New Orleans, USA.

Progress 09/01/13 to 08/31/14

Outputs
Target Audience: Two undergraduate students were offered summer internships in my Lab and worked on robot testing and algorithm development. The PI and co-PIs mentored two PhD and two MSc students in the context of the project; informal laboratory instruction was the key instrument. Our team visited two growers in Watsonville, CA and informed them about the project and conducted field experiments. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided? Two PhD students and three MSc students were mentored by the PI and co-PIs in the context of the project. The PI and one MSc student participated in and presented at the 2014 ASABE International Meeting in Montreal. How have the results been disseminated to communities of interest? The results of this reporting period were presented to the technical and scientific communities at the 2014 ASABE International meeting in Montreal, Canada. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? Our initial findings indicate that it is possible for automated harvest-aid machines to monitor the productivity and ergonomics of strawberry pickers while they are picking. Using this information automated harvest-aids can adapt their operation to serve the pickers faster without compromising their safety. A. First goal: Develop stochastic models of picking and transport operations 1) Major activities and experiments. A mobile sensing system was built and software was developed to collect worker motion data during strawberry picking and data were collected during commercial harvesting. The data will help in developing the stochastic picking model. Also, the task of servicing a number of transport requests in a field was modeled using mixed integer linear programming (MILP). This is a static version of the actual picking process were transport requests are generated continuously, but serves as a starting point for modeling. Finally, a harvest aid robot prototype was designed, built and tested and is undergoing improvements. 2) Data collected. Worker location, motion, posture and picking rate data were collected during commercial harvesting. Data analysis was initiated and will continue in the coming year. Robot navigation in actual fields was also tested and data were collected to assess performance. 3) Discussion of results The collected data indicate that it is possible to monitor picking rate and worker posture in real time using appropriate wearable sensors and an instrumented picking cart. Also, it is feasible to have a small footprint robot navigate in narrow furrows to transport crops. 4) Key accomplishments Development of the data collection system and successful test during commercial harvesting. Development of crop transport model. Development of physical robot prototype. B. Second goal: Integrate models and predictive dispatcher 1) Major activities and experiments. Under the second goal, instances of the developed static crop transport model were solved optimally using a MILP solver. The problem was solved for robot teams of different sizes to investigate the dependence of waiting times on team size. Also, a model of the robot prototype was developed for an open-source robotic simulator (Gazebo). 2) Data collected. No data collection was necessary for this activity. 3) Discussion of results The main result was that the maximum waiting time of pickers for a robot to arrive decreases exponentially as the size of the robot team increases. Of course, the number of robots is limited in practice by cost, maintenance and deployment issues. Hence, more analysis will be performed to come up with reasonable team sizes for different crew sizes. 4) Key accomplishments The optimization of crop transport problems of increasing size.

Publications

• Type: Conference Papers and Presentations Status: Published Year Published: 2014 Citation: Wei-jiunn, J., Lewis, G., Hoachuck, J., Slaughter, D., Wilken, K., Vougioukas, S. (2014). Vibration-reducing Path Tracking Control for a Strawberry Transport Robot. ASABE Annual Intl. Meeting; Paper Number 1914011, Montreal, Quebec, Canada.
• Type: Conference Papers and Presentations Status: Published Year Published: 2014 Citation: Farangis Khosro, A., Rehal, R., Fathallah, F., Wilken, K., Vougioukas, S. (2014). Sensor-based Stooped Work Monitoring in Robot-aided Strawberry Harvesting. ASABE Annual Intl. Meeting; Paper Number 1913911, Montreal, Quebec, Canada.