Source: TANNER RESEARCH, INC. submitted to NRP
AN AUTONOMOUS ROBOT FOR FIELD TRANSPORT OF STRAWBERRIES
Sponsoring Institution
National Institute of Food and Agriculture
Project Status
COMPLETE
Funding Source
Reporting Frequency
Annual
Accession No.
1010094
Grant No.
2016-33610-25691
Cumulative Award Amt.
$599,998.00
Proposal No.
2016-03971
Multistate No.
(N/A)
Project Start Date
Sep 1, 2016
Project End Date
Apr 30, 2019
Grant Year
2016
Program Code
[8.13]- Plant Production and Protection-Engineering
Recipient Organization
TANNER RESEARCH, INC.
825 S MYRTLE AVE
MONROVIA,CA 91016
Performing Department
(N/A)
Non Technical Summary
Project SummaryWe propose to continue the research and development of a four-wheeled autonomous agricultural robot that, while light-weight and affordable, can assist in harvesting by transporting produce from a human picker in the field to a collection station at the edge of the field thus increasing picker productivity by 50%.Our primary focus for this effort will be on the harvesting of strawberries. In California alone, more than 2 billion pounds of strawberries are harvested each year with a value of over $2 billion. The commodity nature of this crop applies pressure on the farmers to keep prices low, yet at the same time labor shortages are growing because picker jobs are unattractive to most Americans. Each picker harvests about $65,000 worth of strawberries per year, so increasing each picker's productivity by 50% allows them to pick an additional $32,000 worth of strawberries per year. Therefore farmers are highly motivated to adopt technology to boost picker efficiency.In the field, pickers do a very complex picking, sorting, and packing task that will take years to automate. However, in the traditional manual harvest scenario, pickers spend about 1/3 of their time transporting a full box of berries along their furrow to the nearest roadway and from there to a nearby station for inspection, payment registration, and stacking on pallets. Offloading this 1/3 non-picking transport time increases the picker's picking time by 50%.We have been working with a nearby strawberry farm, Terry Farms, to define an autonomous robot that can relieve the picker from the transport task to allow them to focus full time on the picking, sorting, and packing task which is their highest value add. The robot must be safe, affordable, and require a minimum of focus from the picker. The robot must be rugged, low power, lightweight, be capable of carrying a 10 pound box of strawberries and be able to operate for hours without recharging. Prior to and during the Phase I effort, we have designed and constructed the hardware and firmware for a prototype that meets these specifications. During Phase I, we began developing the guidance and control software.The complexity of robot autonomy includes image processing, scene understanding, predictive modeling, and path planning. Robot autonomy in a complex and unstructured environment is beyond the scope of an SBIR effort. However, strawberry fields and their surrounding roads are very regular, so the software task is tractable. By the end of Phase I we demonstrated furrow tracking, collision avoidance, picker following, and picker leading capabilities. We propose here to continue the advancement of the control software. By the end of Phase II we will demonstrate autonomous operation including hand-off in the field, obstacle accommodation, collection station identification, hand-off at the collection station, return-to-my-picker, and pack-up-and-go-home behaviors. Phase II will include field trials with multiple robots, each interacting with its own picker team and with the common inspection station.This effort utilizes the expertise at Tanner Research in mechanical design, power electronics, firmware, and software development including image processing, tracking, and advanced algorithms with large data and throughput requirements. For this work, we will augment our internal team with specific robot expertise from our collaborators at Caltech, Prof. Joel Burdick, and his graduate students.We expect to commercialize the results first with our lead customers Terry Farms and Driscoll's, and then, after rapid refinement, expand the roll-out to strawberry farmers statewide and worldwide. We expect to adapt the robot to other applications including transport of other crops, data gathering such as rapid phenotyping, picking, planting, weeding, and pest remediation.
Animal Health Component
100%
Research Effort Categories
Basic
0%
Applied
100%
Developmental
0%
Classification

Knowledge Area (KA)Subject of Investigation (SOI)Field of Science (FOS)Percent
40211222020100%
Knowledge Area
402 - Engineering Systems and Equipment;

Subject Of Investigation
1122 - Strawberry;

Field Of Science
2020 - Engineering;
Goals / Objectives
We propose to continue the research and development of a four-wheeled autonomous agricultural robot that, while light-weight and affordable, can assist in harvesting by transporting produce from a human picker in the field to a collection station at the edge of the field thus increasing picker productivity by 50%.Our primary focus for this effort will be on the harvesting of strawberries. In California alone, more than 2 billion pounds of strawberries are harvested each year with a value of over $2 billion. The commodity nature of this crop applies pressure on the farmers to keep prices low, yet at the same time labor shortages are growing because picker jobs are unattractive to most Americans. Each picker harvests about $65,000 worth of strawberries per year, so increasing each picker's productivity by 50% allows them to pick an additional $32,000 worth of strawberries per year. Therefore farmers are highly motivated to adopt technology to boost picker efficiency.In the field, pickers do a very complex picking, sorting, and packing task that will take years to automate. However, in the traditional manual harvest scenario, pickers spend about 1/3 of their time transporting a full box of berries along their furrow to the nearest roadway and from there to a nearby station for inspection, payment registration, and stacking on pallets. Offloading this 1/3 non-picking transport time increases the picker's picking time by 50%.We have been working with a nearby strawberry farm, Terry Farms, to define an autonomous robot that can relieve the picker from the transport task to allow them to focus full time on the picking, sorting, and packing task which is their highest value add. The robot must be safe, affordable, and require a minimum of focus from the picker. The robot must be rugged, low power, lightweight, be capable of carrying a 10 pound box of strawberries and be able to operate for hours without recharging. Prior to and during the Phase I effort, we have designed and constructed the hardware and firmware for a prototype that meets these specifications. During Phase I, we began developing the guidance and control software.The complexity of robot autonomy includes image processing, scene understanding, predictive modeling, and path planning. Robot autonomy in a complex and unstructured environment is beyond the scope of an SBIR effort. However, strawberry fields and their surrounding roads are very regular, so the software task is tractable. By the end of Phase I we demonstrated furrow tracking, collision avoidance, picker following, and picker leading capabilities. We propose here to continue the advancement of the control software. By the end of Phase II we will demonstrate autonomous operation including hand-off in the field, obstacle accommodation, collection station identification, hand-off at the collection station, return-to-my-picker, and pack-up-and-go-home behaviors. Phase II will include field trials with multiple robots, each interacting with its own picker team and with the common inspection station.This effort utilizes the expertise at Tanner Research in mechanical design, power electronics, firmware, and software development including image processing, tracking, and advanced algorithms with large data and throughput requirements. For this work, we will augment our internal team with specific robot expertise from our collaborators at Caltech, Prof. Joel Burdick, and his graduate students.We expect to commercialize the results first with our lead customers Terry Farms and Driscoll's, and then, after rapid refinement, expand the roll-out to strawberry farmers statewide and worldwide. We expect to adapt the robot to other applications including transport of other crops, data gathering such as rapid phenotyping, picking, planting, weeding, and pest remediation.
Project Methods
The physical robot platform will be fabricated in our laboratory in Monrovia, CA using inexpensive, readily available components when possible (e.g. PVC pipe, electric scoot motors) and augmented with 3D printed plastic parts printed on our 3 commercial and 1 proprietary plastic filament 3D printers. We perform simple testing on the bench with the robot wheels suspended just off the floor. We perform basic functional testing in our parking lot with a constructed sample strawberry beds. And we perform real-world testing at actual strawberry farms of our lead customers that are within an hour drive of our lab. This methodology allows us to quickly iterate improvments to the robot platform by testing at a level appropriate to the improvement, evaluating the results and potentially iterating again.The Phase II effort will focus largely on developing the software required to process the image stream from the robots cameras and implement the more and more sophisticated behavior of the robots. We will continue our software development methodology that allows us to capture field video and data streams and evaluate the performance of the robot in detail back in the lab. As with the hardware platform development, we expect the software development to benefit from the tight loop between development, testing, evaluation, and improvement.

Progress 09/01/18 to 04/30/19

Outputs
Target Audience:Strawberry growers primarily in Southern California. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?Our technical staff continued to attend Nvidia webinars to remain current on the latest development technologies and techniques. How have the results been disseminated to communities of interest?In addition to field demonstrations to strawberry growers, we have had several discussions with the California Strawberry Commission and their Automation Commissioner. We have also updated our corporate website to contain information about our research efforts. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

Impacts
What was accomplished under these goals? Our goal in the Phase II effort was to solve the hardware and software technical problems necessary to bring a low cost, lightweight robot for transporting strawberries to market. During the course of the effort, we removed these barriers and are prepared to use internal funds to transition our designs from a research platform to a commercial product in Phase III. In our Phase II Proposal, we outlined five key tasks in our work plan: Platform Evolution Development Environment & Exception Handling Human-Robot Interaction Navigation Group Behavior In the second half of Phase II from September 2017 to April 2019, we continued working on Task 3 and Task 4 to support the ultimate goal of having coordinated operation between multiple robots (Task 5). We continued our software development for autonomous control and completed Tasks 3 and 4. We have upgraded our computer vision algorithms to work with a multithreaded, object-oriented software framework for efficient operation on an economical embedded system. While progress was made on Task 5, some additional development work will be needed in Phase III. Throughout the research process, we optimized our algorithms for performance and reliability. We also continued to evaluate the reliability of our platform and make any changes as necessary to balance robustness with the economic needs of our growers.

Publications


    Progress 09/01/16 to 04/30/19

    Outputs
    Target Audience:Strawberry growers primarily in Southern California. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?In the first half of the Phase II, three of our technical staff members attended tutorials at the Motor and Drive Systems 2017 conference related to electric motor design and selection. Staff have also attended webinars from Nvidia related to the latest advances in deep learning and neural networks, and how to implement them on Jetson hardware used in our robot. Our key junior mechanical engineer on the project has taken an on-line course in programming in order to make him a more well-rounded roboticist. The prototype robotic platform has also been showcased at several community STEM outreach events from local parades to events with FIRST robotics students. How have the results been disseminated to communities of interest?In addition to field demonstrations to strawberry growers, we have had several discussions with the California Strawberry Commission and their Automation Commissioner. We have also updated our corporate website to contain information about our research efforts. What do you plan to do during the next reporting period to accomplish the goals? Nothing Reported

    Impacts
    What was accomplished under these goals? Our goal in the Phase II effort was to solve the hardware and software technical problems necessary to bring a low cost, lightweight robot for transporting strawberries to market. During the course of the effort, we removed these barriers and are prepared to use internal funds to transition our designs from a research platform to a commercial product in Phase III. In our Phase II Proposal, we outlined five key tasks in our work plan: Platform Evolution Development Environment & Exception Handling Human-Robot Interaction Navigation Group Behavior During the first half of Phase II for the period spanning September 2016 to September 2017, we made significant progress of Tasks 1 & 2. We evolved our platform to use more powerful motors on a 24V system and doubled the effective torque of our steer motors. Our platform is sufficiently robust to serve as a development platform for the computer vision. With the addition of the Nvidia Jetson TX1, we have a more powerful and efficient embedded computer (62% reduction in peak power consumption). As part of Task 2, we have restructured our code base and incorporated a state machine as a logical framework for world modeling and event handling. As of April 2019, the code base contained over 14,000 lines of code (not including 3rd party libraries, such as OpenCV) running on our embedded computer and microcontroller. We also made significant progress on Tasks 3 and 4 and demonstrated the robot's ability to navigate out of the furrow towards a collection station. Along the way, we are using a HOG +SVM Pedestrian detector to recognize humans in the path of the robot. Our robot's stereo camera creates a disparity map that can be used to build a 3D model of the scene for collision avoidance. In the second half of Phase II from September 2017 to April 2019, we continued working on Task 3 and Task 4 to support the ultimate goal of having coordinated operation between multiple robots (Task 5). We continued our software development for autonomous control and completed Tasks 3 and 4. While progress was made on Task 5, some additional development work will be needed in Phase III. Throughout the research process, we optimized our algorithms for performance and reliability. We also continued to evaluate the reliability of our platform and make any changes as necessary to balance robustness with the economic needs of our growers. At the completion of the Phase II research, we have an efficient robotic software stack that contains the key elements necessary to transport strawberries from the picker to a collection station. While there is still a little bit of software integration work to complete, we are prepared to do so with internal funds to bring the product to market in time for the harvest in the 2020 season. We are thankful for the support and feedback we have received from the USDA, the California Strawberry Commission, and the growers we have worked with during this effort.

    Publications


      Progress 09/01/17 to 08/31/18

      Outputs
      Target Audience:Strawberry growers primarily in Southern California. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?Our key junior mechanical engineer on the project has taken an on-line course in programming in order to make him a more well-rounded roboticist. How have the results been disseminated to communities of interest?Field demonstrations to strawberry growers leading the way in automation. What do you plan to do during the next reporting period to accomplish the goals?We will continue our software development for autonomous control to complete the remainder of tasks specified in our Phase II proposal. We will complete the operation of returning from the collection station, entering the furrow, and returning to working with the picker. Throughout this development, we will optimize our algorithms for performance and reliability. We will establish rules and protocols for fleet operation. We will also continue to evaluate the reliability of our platform and make any changes as necessary to balance robustness with the economic needs of our growers.

      Impacts
      What was accomplished under these goals? During the second year of the Phase II effort, we have focussed on developing the software to process video data from the field and extract from the video stream a representation of the 3D world sufficient to perform the required behaviors of steering, navigation, following, and collision avoidance. Our early algorithms used simplistic raster scan techniques, but since then we have progressed to algorithms incorporating classic morphology, line tracing, correlation coefficients, and stereo camera stereopsis. More detail on this work will be in the technical report.

      Publications


        Progress 09/01/16 to 08/31/17

        Outputs
        Target Audience:Strawberry growers primarily in Southern California. Changes/Problems: Nothing Reported What opportunities for training and professional development has the project provided?Three of our technical staff members attended tutorials at the Motor and Drive Systems 2017 conference related to electric motor design and selection. Also staff have attended webinars from nVidia related to the latest advances in deep learning and neural networks, and how to implement them on Jetson hardware such as used in our robot. How have the results been disseminated to communities of interest?Field demonstrations to strawberry growers leading the way in automation. What do you plan to do during the next reporting period to accomplish the goals?We will continue our software development for autonomous control to complete the remainder of Tasks 3, 4, and 5. In the next half of Phase II, we will complete the operation of returning from the collection station, entering the furrow, and returning to working with the picker. Throughout this development, we will optimize our algorithms for performance and reliability. We will establish rules and protocols for fleet operation. We will also continue to evaluate the reliability of our platform and make any changes as necessary to balance robustness with the economic needs of our growers.

        Impacts
        What was accomplished under these goals? In our Phase II Proposal, we outlined five key tasks in our work plan: Platform Evolution Development Environment & Exception Handling Human-Robot Interaction Navigation Group Behavior During the first half of Phase II for the period spanning September 2016 to September 2017, we have made significant progress of Tasks 1 & 2 and are in active development of Tasks 3 & 4 to lay the ground work for Task 5. For Task 1, we have evolved our platform to use more powerful motors on a 24V system and doubled the effective torque of our steer motors. Our platform is sufficiently robust to serve as a development platform for the computer vision. With the addition of the Nvidia Jetson TX1, we have a more powerful and efficient (62% reduction in peak power consumption) embedded computer. As part of Task 2, we have restructured our code base and incorporated a state machine as a logical framework for world modeling and event handling. As of August 31st, the code base contained over 10,000 lines of code (not including 3rd party libraries, such as OpenCV) running on our embedded computer and microcontroller. We also made significant progress on Tasks 3 and 4 and demonstrated the robot's ability to navigate out of the furrow towards a collection station. Along the way, we are using a HOG +SVM Pedestrian detector to recognize humans in the path of the robot. Our robot's stereo camera creates a disparity map that can be used to build a 3D model of the scene for collision avoidance.

        Publications