Progress 12/01/16 to 11/30/21
Outputs
Target Audience:The RAPID project targets an heterogenous audience. Within academia, RAPID aims at tackling both basic and applied science and technology (in particular robotics, mechatronics, and artificial intelligence). Consequently, RAPID related outcomes have been described in various scientific papers submitted to leading international robotics conferences and journal sponsored by the IEEE Robotics and Automation Society (IEEE RAS). Conferences include the IEEE International Conference on Robotics and Automation (ICRA), the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), the IEEE International Conference on Automation Science Engineering (CASE). RAPID related findings have been published in journals such as the IEEE Transactions on Automation Science and Engineering, the IEEE Transactions on Robotics, the IEEE Robotics and Automation Letters, and Remote Sensing. A second target audience include farmers and in particular winegrowers. E&J Gallo winery has been a key partner throughout the project by providing domain expertise and allowing access to their vineyards. Other growers (e.g., almond growers) have also been engaged through the periodic roundtables on ag-food-tech hosted by CITRIS (Center for Information Technology Research in the Interest of Society) at UC Merced. Non-technical articles about RAPID also appeared in magazines targeting farmers, such as the "Irrigation" magazine. The third segment in the target audience includes tech entrepreneurs or corporate entities that may be interested in entering the sector of ag-tech. Throughout the duration of the project, presentations have been given in venues such as CES in Las Vegas, the Deep Learning Summit in San Francisco, and the California Food Processing Expo in Santa Clara, just to name a few. The final target includes the general population, in an effort to raise awareness about the necessity to create a resilient food-supply chain. To this end, various non-technical presentations have been given in outlets meant to reach this target audience. For example, the RAPID project was featured in the popular "Science Friday" show on NPR (episode "Eating Smarter in a Smarter World" on 07/12/2019). Changes/Problems:The later part of the project has been severely impacted by the emergence of the COVID pandemic and the consequent restrictions put in place by our institutions to ensure the health and safety of students. Starting from March 2020 experimental activities (and in particular field experiments) came to an almost complete stop. Despite our expectation to resume activities in 2021, the emergence of the Delta variant forced us to take an extremely prudent approach to research restart. Consequently, we performed more computational and lab work than anticipated, but less field validation. The upshot to this unexpected problem is that RAPID has been instrumental to generate the momentum for other related activities, and therefore this line of research will continue also after the project ended. What opportunities for training and professional development has the project provided?During this project we have involved, undergraduates, graduate students, and postdoctoral scholars as the participating campuses. Due to the pandemic, undergraduate student involvement decreased since March 2020. Research done by graduate students as part of this project has been integral to their doctoral dissertation. At UC Merced, PhD student Thomas Thayer worked on scheduling program throughout the duration of this grant and lead the development of the software stack controlling the robot for data acquisition in the vineyard (he graduated in July 2020 and is now working as robotics engineer at the SLAC National Accelerator Laboratory in Stanford). Various undergraduate students and a MS student at UC Merced assisted in this project with data collection in the field, doing minor hardware development (e.g., integrating a soil moisture probe into the robot for autonomous measurement), as well as coding in and data post-processing. At UC Berkeley, graduate, undergraduate and postdoctoral personnel worked on a variety of topics related to the goals described above. In particular, they designed, prototyped, developed and tested robotic grippers and the associated software to autonomously adjust variable rate emitters, and also developed software estimate soil moisture content from remote sensing and imaging. Students involved in these activities co-authored all the scientific publications appeared during this project, with some including personnel from the different campuses. How have the results been disseminated to communities of interest?During this project we have produced: 16 conference papers appeared in leading international conferences; 6 journal articles; 1 book chapter. Full details about these publications have been provided under the "Products Section." In addition, numerous talks describing some of the research performed as part of RAPID have been given in "non-academic" venues with the objective of enhancing public understanding and increasing interest in how robotics and artificial intelligence can be leveraged in food production. Examples of activities targeting the general public include a segment on NPR (Science Friday on 7/12/2019), a keynote talk at the IEEEConference on Technologies for Sustainability (October 2018), and the Maker Faire in Rome (December 2020). What do you plan to do during the next reporting period to accomplish the goals?
Nothing Reported
Impacts
What was accomplished under these goals?
In the following we provide details about the results obtained for each of the four goals identified in the beginning of the project. All results described in the following are presented in the papers listed under the "Products" section. Design of low-cost, robust adjustable emitters and portable adjusting devices. Our starting objective was retrofitting existing systems with low-cost, passive, plastic, screw-adjustable emitters that are commercially available. Accordingly, as initial prototype we developed a hand-held device that would allow passive emitters to be systematically adjusted in the field by human and robot teams to fine-tune water delivery at the plant level. Its coarse-to-fine mechanism to facilitates alignment to passive emitters in the field and precise automated adjustment of flow settings. Preliminary results confirmed the viability of this approach. In a second design iteration, we designed a lightweight, modular Emitter Localization Device (ELD) with cameras and LEDs that can be non-invasively mounted on a robotic arm. Adjusting emitters uses a two-phase procedure: 1) aligning the robot base using the build-in hand camera, and 2) aligning the gripper axis with the emitter axis using the ELD. We studied success rates and sensitivity analysis to tune computer vision parameters and joint motor gains. Experiments suggest that emitters can be adjusted with 95% success rate in approximately 20 seconds. ELD was mounted and tested on a Toyota HSR mobile manipulator robot that autonomously adjusted low-cost passive emitters. Data- and model-driven irrigation scheduling. Recent advances in unmanned aerial vehicles suggest that collecting aerial agricultural images can be cost efficient, which can subsequently support automated precision irrigation. To study the potential for machine learning to learn local soil moisture conditions directly from such images, we developed a very fast, linear discrete-time simulation of plant growth based on the Richards equation. We used the simulator to generate large datasets of synthetic aerial images of a vineyard with known moisture conditions and then compare seven methods for inferring moisture conditions from images, in which the "uncorrelated plant" methods look at individual plants and the "correlated field" methods look at the entire vineyard: 1) constant prediction baseline, 2) linear Support Vector Machines (SVM), 3) Random Forests Uncorrelated Plant (RFUP), 4) Random Forests Correlated Field (RFCF), 5) two-layer Neural Networks (NN), 6) Deep Convolutional Neural Networks Uncorrelated Plant (CNNUP), and 7) Deep Convolutional Neural Networks Correlated Field (CNNCF). Experiments on held-out test images show that a globally connected CNN performs best with normalized mean absolute error of 3.4%. Sensitivity experiments suggest that learned global CNNs are robust to injected noise in both the simulator and generated images as well as in the size of the training sets. In simulation, we compared the agricultural standard of flood irrigation to a proportional precision irrigation controller using the output of the global CNN and find that the latter can reduce water consumption by up to 52% and is also robust to errors in irrigation level, location, and timing. Workload scheduling: despite the increased availability of robots in the field, these devices continue to remain a scarce resource and we therefore studied different algorithms to solve the optimization problem to determine which subset of tasks (emitter adjustment) should be tackled by the robots under the constrained of limited operation time due to energy constraints. We formulated this problem as an instance of a classic optimization problem known as orienteering. Orienteering is a hard computational problem that cannot be efficiently solved exactly. Therefore, heuristic solutions are needed. An additional peculiar aspect of the problem at hand is that motion in a vineyard is constrained by the presence of trellis and irrigation line, thus generating motion constraints commonly not addressed in literature. Against this backdrop, we developed a suite of algorithms solving different instances of this problem. In the simplest case where all known quantities are known without uncertainty, we first studied the case where a single agent is used, and then the case where multiple agents coordinate to jointly solve an assigned set of tasks. This latter version can also accommodate heterogenous agents, e.g., humans and robots with different capabilities. Notably, this last line of work won the best paper award at the 2018 IEEE International Conference on Automation Science and Engineering. Stimulated by the fact that a robot can two multiple tasks at once when out in the field (e.g., adjust an emitter and take a soil moisture sample), we also considered a new orienteering problem where the robot must optimize two sets of tasks at the same time. To the best of our knowledge this problem had never been studied before. Finally, we considered the significantly harder version of the problem known as stochastic orienteering where some of the quantities are not exactly known. Specifically, we considered the case where the time spent to move between different locations in the field is known exactly known but is rather a random variable. For this problem we developed a novel suite of algorithms based on constrained Markov Decision Processes that do not produce a path, but rather a policy that adapts to realization of the random variables describing travel times. All these algorithms have been tested and validated in datasets collected in vineyards managed by our commercial partner. Field and laboratory experiments: field experiments were mostly conducted in the early part of the project because the COVID pandemic prevented field work in the later part of the project. We collected aerial imagery and soil moisture measurement in two commercial vineyards and we also concurrently tested in the field our robot navigation stack. In particular, in summer 2018 we collected aerial imagery over a vineyard for an entire growing season, thus providing a baseline to evaluate our algorithms. This study compared measurements of evapotranspiration (ET) from using data collected from drones and earth observation system sources for 10 events over a growing season using multiple ET estimation methods. Results indicated that drone ET estimates that include non-canopy pixels are generally lower on average than remote sensing methods. This study indicates that limited deployment of drones can provide important estimates of uncertainty in remote sensing ET estimations for larger areas and to also improve irrigation management at a local scale. With regard to laboratory experiments, we designed and developed AlphaGarden: an autonomous polyculture garden that prunes and irrigates living plants in a 1.5m × 3.0m physical testbed. AlphaGarden uses an overhead camera and sensors to track the plant distribution and soil moisture. We modeled individual plant growth and interplant dynamics to train a policy that chooses actions to maximize leaf coverage and diversity. For autonomous pruning, AlphaGarden uses two custom-designed pruning tools and a trained neural network to detect prune points. We present results for four 60-day garden cycles. Results suggest AlphaGarden can autonomously achieve 0.96 normalized diversity with pruning shears while maintaining an average canopy coverage of 0.86 during the peak of the cycle.
Publications
- Type:
Journal Articles
Status:
Published
Year Published:
2021
Citation:
T. Thayer, S. Carpin. "An Adaptive Method for the Stochastic Orienteering Problem". In IEEE Robotics and Automation Letters, 6(2):4185-4192, 2021
- Type:
Journal Articles
Status:
Published
Year Published:
2021
Citation:
X. Kan, T. Thayer, S. Carpin, K. Karydis. "Task planning on stochastic aisle graphs for precision agriculture." In IEEE Robotics and Automation Letters, 6(2):3287-3294
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2021
Citation:
T. Thayer, S. Carpin. "A Resolution Adaptive Algorithm for the Stochastic Orienteering Problem with Chance Constraints" Proceedings of the 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, 6388-6395
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2021
Citation:
T. Thayer, S. Carpin. "A Fast Algorithm for Stochastic Orienteering with Chance Constraints." Proceedings of the 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, 7398-7945
- Type:
Journal Articles
Status:
Awaiting Publication
Year Published:
2022
Citation:
Y.Avigal, W.Pong, M. Presten, S. Aeron, A. Deza, S. Sharma, R. Parikh, S. Oehme, S. Carpin, J. Viers, S. Vougioukas, K. Goldberg. "Simulating Polyculture Farming to Learn Automation Policies for Plant Diversity and Precision Irrigation". IEEE Transactions on Automation Science and Engineering
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2021
Citation:
Y. Avigal, A. Deza, W. Wong, S. Oehme, M. Presten, M. Theis, J. Chui, P. Shao, H. Huang, A. Kotani, S. Sharma, M. Luo, S. Carpin, Joshua H. Viers, S. Vougioukas, K. Goldberg. "Learning Seed Placements and Automation Policies for Polyculture Farming with Companion Plants". Proceedings of the 2021 IEEE International Conference on Robotics and Automation, 902-908
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2021
Citation:
Simeon Adebola, Ken Goldberg. "Assisting Polyculture Farming in Africa".
IEEE AfriCon, 1-2.
- Type:
Journal Articles
Status:
Published
Year Published:
2022
Citation:
F. Betti Sorbelli, S. Carpin, F. Cor�, S.K. Das, A. Navarra, C.M. Pinotti. Speeding up Routing Schedules on Aisle-Graphs with Single Access. In IEEE Transactions on Robotics, 38(1):433-447.
|
Progress 12/01/19 to 11/30/20
Outputs
Target Audience:The RAPID project is relevant to an heterogenous audience. Within academia, RAPID aims at tackling both basic andapplied science and technology. Consequently, RAPID related outcomes have been described in various papers submittedto leading international robotics conferences sponsored by the IEEE Robotics and Automation Society (IEEE RAS). Vine-growers continue to be engaged as part of the RAPID project (in particular E&J Gallo winery continues to be a key partner inthe project). The RAPID project have also been featured in publications specifically targeting farmers and growers. For example, in October 2020 an article describing our progrject appeared in AgAlert, "the weekly newspaper for California agriculture" (https://www.agalert.com/story/?id=14359). In the last year, presentationas about the RAPID project have also been given at various non-academic AI and Robotics meetings such as CES in Las Vegas, the Deep Learning Summit in San Francisco, adn the California Food Processing Expo in Santa Clara, just to name a few. Overall, 8 such presentations were given during the current reporting period. Changes/Problems:Due to the ongoing pandemic and the restrictions put in place by our institutions to ensure the health and safety of students, staff, and faculty, our collective resaerch efforts have significantly slowed down since mid March and are resuming only now, but at a much slower pace. In particular, over summer it has been impossible to go out in the field and perform any experimental work. Consequently, we are asking for a one year no-cost extension to allow us to perform in Summer 2021 the field tests we anticipated doing in Summer 2020. What opportunities for training and professional development has the project provided?We have continued to involve various undergraduate students, graduate students, and postdoctoral scholars at UC Merced and UC Berkeley, although the becuase of the pandemic we could not involve undergraduate students during summer as we did in the past. Graduate and undergraduate students have worked in different topics. At UC Berkeley, a group of undergraduate, graduate and postgraduate researchers has continued the development and improvement of a testbed to systematically study how vision based machine learning can be used to understand and predict plant growth and correspondingly adjust water input. In particular a system called AlphaGardenSim has been developed to perform first order simulation of plant growth, extending previous results aiming at using artificial intelligence (and in particular deep neural networks) to predict the plants' water intake. The testbed has then extended in a physical farming testbed where learning algorithms have been trained to learn a variety of tasks related to irrigation, pruning, and other ag-relevant activities.At UC Merced, students involved in the project have continued to work on scheduling problems aimed at optimizing route scheduling for the robot operating in the vineyard. How have the results been disseminated to communities of interest?Results have been widely disseminated, as also indicated in the "Target Audience" section. Papers were submitted and published in the proceedings of the leading robotics and automation conferences sponsored by the IEEE Robotics and Automation Society, such as the International Conference on Robotics and Automation (ICRA), the International Conference on Intelligent Robots and Systems (IROS) and the Interanational Conference on Automation Science and Engineering (CASE). Two papers also appeared in international journals, including Remote Sensing. Notably, the paper appeared in the CASE conference won the best student paper award. As pointed out in the target audience section, various presentations have also been given at non-academic venues focussing on technology and artificial intelligence. A website has also been setup to make our findings easily accessible (http://rapid.berkeley.edu/). What do you plan to do during the next reporting period to accomplish the goals?Given the nature of the project and the relatively short time span during which vineyards are irrigated (essentially April to August), we had asked a a no-cost one year extension to take advantage of an additional season of data and validation. The objective for the year just concluded was to demonstrate the integration of all components developed thus far. Leveraging our partnership with E&J Gallo, we are exploring whether it is possible to retrofit part of the existing irrigation infrastructure with variable rate emitters that can be adjusted by our custom-designed actuator, or whether it is possible to install additional irrigation lines specifically dedicated to testing our project. However, as pointed out in the next section, because of the pandemic we had to stop all our field activities. Hoping in a further no-cost-extension, the plan for next year would be to complete what we have not been able to do this year because of the COVID-19 disruption.
Impacts
What was accomplished under these goals?
Most of the goals we identified in the initial proposal have been accomplished in "individual modules." We have developed and tested mechanisms to latch to variable rate emitters and make adjustments, data-driven algorithms to infer relevant parameters from data, and algorithms to schedule the tasks of the robots in the field in a way that is compatible with their energy constraints Extensive small scale laboratory setups have been developed. Our next objective is to tie all these partial results together in a compresensive system that can demonstrate all functionalities at once.
Publications
- Type:
Journal Articles
Status:
Published
Year Published:
2020
Citation:
T. Thayer, S. Vougioukas, K. Goldberg, S. Carpin. Multi-robot Routing Algorithms for Robots Operating in Vineyards. In IEEE Transactions on Automation Science and Engineering, 17(3):1184-1194, 2020
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2020
Citation:
T. Thayer, S. Carpin. "Solving Large-Scale Stochastic Orienteering Problems with Aggregation ", Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2452-2458
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2020
Citation:
F. Betti Sorbelli, S. Carpin, F. Coro', A. Navarra, C.M. Pinotti. "Optimal Routing Schedules for Robots Operating in Aisle-Structures". Proceedings of the 2020 IEEE International Conference on Robotics and Automation, 4927-4933
- Type:
Journal Articles
Status:
Published
Year Published:
2020
Citation:
M. Kalua, A.M. Rallings, L. Booth, J. Medellin-Azuara, S. Carpin, J. Viers. "sUAS Remote Sensing of Vineyard Evapotranspiration Quantifies Spatiotemporal Uncertainty in Satellite-Borne ET Estimates." Remote Sensing, 12(19):3251, 2020
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2020
Citation:
Yahav Avigal, William Wong, Jensen Gao, Kevin Li, Mark Theis, Mark Preston, Grady Pierroz, Fang Shuo Deng, Ken Goldberg. "Simulating Polyculture Farming to Tune Automation Policies for Plant Diversity and Precision Irrigation." Proceedings of the IEEE Conference on Automation Science and Engineering, 238-245, 2020
|
Progress 12/01/18 to 11/30/19
Outputs
Target Audience:The RAPID project is relevant to an heterogenous audience. Within academia, RAPID aims at tackling both basic and applied science and technology. Consequently, RAPID related outcomes have been described in various papers submitted to leading international robotics conferences sponsored by the IEEE Robotics and Automation Society (IEEE RAS). Vine-growers continue to be engaged as part of the RAPID project (in particularE&J Gallo winery continues to be a key partner in the project). The RAPID project has also been featured on the 7/12/2019 episode of the radio show "Science Friday," a show typically followed by more than one million listeners from all over the US (https://www.sciencefriday.com/segments/eating-smarter-in-a-warming-world/). Changes/Problems:As agreed with the program director, a no-cost extension has been negotiated to extend the experimental part of the project. What opportunities for training and professional development has the project provided?During this second year, we have continued to involve various undergraduate students, graduate students, and postdoctoral schoalrs at UC Merced and UC Berkeley. Trainees have been involved in a variety of activities, from algorithmic design to experimental activities involving field work and hardware design. At UC Merced, one graduate student has continued to work on scheduling algorithms (workload scheduling) considering a novel extension with multiple goal whereby the robot aims at both visiting locations of interest for emitter adjustment as well as locations where "interesting" soil moisture samples can be collected. Undergraduate students have also been involved in multiple aspects. One student has adapted formerly developed convolutional neural networks to process field data collected during summer 2018. This work has outlined strengths and limits of the proposed approach. The same student has also perfected the control software allowing the robot to autonomously navigate in vineyards. Another undergraduate student has developed an inexpensive robot arm to be mounted on the robot as an alternative design to more sophisticated commercial options. At UC Berkeley, a group of undergraduate, graduate and postgraduate researchers has developed a testbed to systematically study how vision based machine learning can be used to understand and predict plant growth and correspondingly adjust water input. How have the results been disseminated to communities of interest?Results have been widely disseminated, as also indicated in the "Target Audience" section. Papers were submitted and published in the proceedings of the IEEE International Conference on Automation Science and Engineering. A book chapter also appeared in a monograph specifically devoted to "Robotics and Automation for Improving Agriculture." In October 2019, the project was also presented at the CITRIS ag-food tech roundtable at UC Merced, a forum attended by farmers, entrepreneurs and investors active in the precision-ag space. Notably, the RAPID project was featured in the radio show "Science Friday," a long-running radio talk show that is weekly followed by more than one million listeners. This opportunity brougth the project to the attention of the broader public beyond academia and growers. A website has also been setup to make our findings easily accessible (http://rapid.berkeley.edu/). What do you plan to do during the next reporting period to accomplish the goals?Given the nature of the project and the relatively short time span during which vineyards are irrigated (essentially April to August), we have asked a a no-cost one year extension to take advantage of an additional season of data and validation. The objective for the final year is to demonstrate the integration of all components developed thus far. Leveraging our partnership with E&J Gallo, we are exploring whether it is possible to retrofit part of the existing irrigation infrastructure with variable rate emitters that can be adjusted by our custom-designed actuator, or whether it is possible to install additional irrigation lines specifically dedicated to testing our project.
Impacts
What was accomplished under these goals?
During the third year of the project we have made significant progress on all areas identified as major goals of the project. We have continued to refine various designs to study the automation of plant-level precision irrigation, specifically learning-based irrigation controllers.To this end, we developed MOLT, a Mess-scale Open-source Low-cost Testbed for robot assisted precision irrigation delivery. The platform is being evaluated using a variety of indoor crops, thus greatly lowering the hurdles related to experimental data collection at scale or with high frequency. We have also designed and implemented an additional robot arm to collect soil moisture samples from the robot and perfected the software to schedule the points of interest where samples should be collected. With regard to workload scheduling, we have identified and defined a new problem, i.e., multi-objective scheduling. This problem arises when the robot is routed through the vineyard in an attempt to obtain two objectives at once, i.e., visiting locations where emitters must be adjusted, as well as locations where soil sample measures should be taken. Indeed, vine growers expressed great interest in using the system we developed to automate the process of "in-situ" soil moisture sample collection. The algorithms we developed allow the user to specify how much effort (defined as energy consumption) should be devoted to either task, and accordingly computes a route respecting the vineyard topology. Finally, the formerly developed convolutional neural networks have been adapted and tested on the datasets collected during Summer 2018. This activity has outlined strengths and limitations of the proposed approach. In particular it has been shown that the architectures we identified though simulated data can be re-used on real world data (after re-training). While the accuracy is lower, as expected given the less discriminative inputs, the approach still method still manages to correctly identify regions that are more humid or dry (as determined by manually collected soil moisture samples).
Publications
- Type:
Book Chapters
Status:
Published
Year Published:
2019
Citation:
S. Carpin, K. Goldberg, S. Vougioukas, R. Berenstein, J. Viers. "Use of intelligent/autonomous systems in crop Irrigation". In Robotics and Automation for Improving Agrilculture, J. Billingsley (Ed.), Burleigh Dodds, pp.137-159.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2019
Citation:
Marius Wiggert, Leela Amladi, Ron Berenstein, Stefano Carpin, Joshua Viers, Stavros Vougioukas, Ken Goldberg. "RAPID-MOLT: A Meso-scale, Open-source, Low-cost Testbed for Robot Assisted Precision Irrigation and Delivery." Proceedings of the IEEE International Conference on Automation Science and Engineering, 1489-1496.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2019
Citation:
T.C. Thayer, S. Vougioukas, K. Goldberg, S. Carpin. "Bi-Objective Routing for Robotic Irrigation and Sampling in Vineyards." Proceedings of the 2019 IEEE International Conference on Automation Science and Engineering, 1481-1488
- Type:
Other
Status:
Published
Year Published:
2019
Citation:
Robots and the Return to Collaborative Intelligence (Commentary). Ken Goldberg. Nature Machine Intelligence Journal. volume 1, pages 24. January 2019
|
Progress 12/01/17 to 11/30/18
Outputs
Target Audience:The RAPID project is relevant to an heterogenous audience. Within academia, RAPID aims at tackling both basic and applied science and technology. Consequently, RAPID related outcomes have been described in various papers submitted to leading international robotics conferences sponsored by the IEEE Robotics and Automation Society (IEEE RAS). Vine-growers continue to be engaged as part of the RAPID project. Through our continued relationship with E&J Gallo winery we have obtained access to an experimental vineyard located in Ripperdan (CA), and have collected a massive dataset that is being shared with them (aerial images, soil moisture data and more). Results related to the RAPID project were also presented at the yearly Merced Ag Tech Fair (March 2018), an event attended by a diverse audience including farmers, community leaders, entrepreneurs, and the general public (about 100 participants). More than 15 lectures/seminars discussing the RAPID project were given nationally and internationally, both at public events and in academic settings. The RAPID project has also been featured in one of the leading online IEEE publications, i.e., "The Institute." (http://theinstitute.ieee.org/technology-topics/robotics/robots-to-help-californias-grape-growers-conserve-water). The online publication is accessible to the all IEEE members, i.e., more than 400,000 students and professionals. Finally, upon invitation, we have published an articole in the Irrigation Today magazine (https://www.irrigation.org/IA/News/Irrigation-Today.aspx). According to their website, "[the] magazinehas a printed distribution to over 13,000 growers, IA member manufacturers and distributors, qualified nonmembers, and industry stakeholders." The online version is also distributed to additional entities (e.g., government agencies, irrigation districts and more) for a total distribution exceeding 29,000. Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?During this second year, we have continued to involve various undergraduate studens, graduate students, and postdoctoral schoalrs at UC Merced and UC Berkeley. At UC Merced we have also involved, and partially supported, a technician in charge of operating the drone. Trainees have been involved in a variety of activities. In particular, undergraduate students at UC Merced have supported data collection in the field, have helped in developing and testing software for autonomous robot operation in vineyards, and have assisted in postprocessing collected data. One undergraduate has also been involved in outreach activities. Graduate students at UC Merced have worked in software and algorithmic developent for autonomous robot operation in vineyards. At UC Berkeley, undergraduate and graduate students have contributed to the development and perfecting of a deep convolutional neural network (CNN) that can learn the inverse mapping between aerial images and soil moisture levels. Postdoctoral fellows have instead worked on the developent of various designs for the robotic gripper. It is worthwhile mentioning that some of the trainees are from under-represented minority groups. How have the results been disseminated to communities of interest?Results have been widely disseminated, as also indicated in the "Target Audience" section. In particular, in 2018 four papers have appeared in leading international robotics and automation conferences sponsored by the IEEE Robotics and Automation Society. Notably, one of them won the Best Conference Paper Award at the 2018 IEEE International Conference on Automation Science and Engineering (top paper out of 430 submissions). The paper describes how to efficiently create routes for mutiple robots autonomously operating in a vineyeard environment (see "Products" section for full citations). In addition, project participants have given more that 15 seminars/lectures regarding the RAPID project, both nationally and internationally. Our project was also noticed by non-academic publishers. In particular, the IEEE featured it on its "The institute" online publication (accessible to more than 400,000 professionals and students), and we were invited to write an article for the "Irrigation Today" magazine (printed and online) that targets growers, irrigation professionals and other relevant stakeholders. A website has also been setup to make our findings easily accessible (http://rapid.berkeley.edu/). What do you plan to do during the next reporting period to accomplish the goals?In the tentative third year extension, all separate components developed so far will be integrated and tested in the field. Leveraging our partnership with E&J Gallo, we are exploring whether it is possible to retrofit part of the existing irrigation infrastruture with variable rate emitters that can be adjusted by our custom-designed actuator, or whether it is possible to install additional irrigation lines specifically dedicated to testing our project. Morevover, irrigation schedules will be informed by our data-driven system. Given the nature of the project and the relatively short time span during which vineyards are irrigated (essentially April to August), we anticipate asking for a no-cost one year extension to take advantage of an addtional season of data and validation.
Impacts
What was accomplished under these goals?
During the second year we have made significant progress on all the four areas described as major goals of the project. Results have been published and disseminated in a variety of ways, as described later on. In the following we briefly outline progress in each area. 1 - we have refined our initial gripper design by coming up with a completely new approach. We designed a lightweight, modular Emitter Localization Device (ELD) with cameras and LEDs that can be non-invasively mounted on a robotic arm. This prototype has been mounted on a commercial arm on a mobile platform and we have developed a control software that can be used by the robot to visually navigate towards an emitter, latch, and adjust it. This approach has been extensively tested in lab conditions and will be next moved to an outdoor domain. 2- with regard to data-driven irrigation scheduling we have developed a new novel method based on convolutional neural network to learn the mapping between aerial images and soil moisture values. Tested for the moment in simulation only, we contrasted this with a variety of methods formerly proposed in literature: 1) constant prediction baseline, 2) linear Support Vector Machines (SVM), 3) Random Forests Uncorrelated Plant (RFUP), 4) Random Forests Correlated Field (RFCF), 5) two-layer Neural Networks (NN). We developed two alternative CNN architectures that clearly outperform all formerly proposed solutions. Based on the predictions provided by the CNN, we then studied how to adjust irrigation using a first order simulation based on Richard's equation. In simulation, we compare the agricultural standard of flood irrigation to a proportional precision irrigation controller using the output of the global CNN and find that the latter can reduce water consumption by up to 52% and is also robust to errors in irrigation level, location, and timing. 3- for what concerns scheduling, we have developed a variety of routing algorithms for single and multiple robots operating in a vineyard environment, i.e., constrained in their motion. The algorithms are an instance of a well known combinatorial problem known as "orienteering" and ask how to adjust as many emitters as possible subject to a constraint on the traveled distance. We have studied both a single and a multi-agent version of the problem that can be adapted to the case where heterogeneous agents cooperate (e.g., humans and robots). Our algorithms clearly outperform formerly proposed heuristics, scale very favorably with the size of the problem, and make efficient use of the allocated budget. 4- building upon the experience gained during summer 2017, in spring/summer 2018 we completed 10 field trips to perform data collection on an experimental vineyard located in Ripperdan (CA) and managed by E&J Gallo. The vineyard is approximately 85 acres. In each field trip, two teams operated in parallel. The first team manually collected soil moisture samples at 72 locations equally spaced in the ranch. Such samples, were later on used to produce a continuous soil moisture map with a kriging algorithm. The second team flew a fixed wing drone over the same vineyard. During each trip a fixed wing drone was flown over the same vineyard collecting a variety of imagery data, including RGB, multispectra, and also LIDAR. Imagery collected with the drone has been post-processed to yield orthorectified image stacks of the block. This massive dataset will then be used to re-train and update the CNNs developed in point 2.
Publications
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2018
Citation:
T. Thayer, S. Vougioukas, K. Goldberg, S. Carpin. "Routing Algorithms for Robot Assisted Precision Irrigation." Proceedings of the 2018 IEEE International Conference on Robotics and Automation, 2221-2228.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2018
Citation:
R. Berenstein, R. Fox, S. McKinley, S. Carpin, K. Goldberg. "Robustly Adjusting Indoor Drip Irrigation Emitters with the Toyota HSR Robot." Proceedings of the 2018 IEEE International Conference on Robotics and Automation, 2236-224.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2018
Citation:
T. Thayer, S. Vougioukas, K. Goldberg, S. Carpin. "Multi-Robot Routing Algorithms for Robots Operating in Vineyards." Proceedings of the 2018 IEEE International Conference on Automation Science and Engineering, 7-14.
- Type:
Conference Papers and Presentations
Status:
Published
Year Published:
2018
Citation:
D. Tseng, D. Wang, C. Chen, L. Miller, W. Song, J. Viers, S. Vougioukas, S. Carpin, J. Aparicio Ojea, K. Goldberg. "Towards Automating Precision Irrigation: Deep Learning to Infer Local Soil Moisture Conditions from Synthetic Aerial Agricultural Images". Proceedings of the 2018 IEEE International Conference on Automation Science and Engineering, 284-291.
- Type:
Other
Status:
Published
Year Published:
2018
Citation:
S. Carpin, K. Goldberg, S. Vougioukas, J. Viers. Using robotics to conserve water. Irrigation Today, 3(1): 8-9, 2018.
- Type:
Websites
Status:
Published
Year Published:
2018
Citation:
http://rapid.berkeley.edu/
|
Progress 12/01/16 to 11/30/17
Outputs
Target Audience:The RAPID project is relevant to a diverse audience. Within academia, RAPID aims at tackling both basic and applied science and technology. Consequently, RAPID related outcomes have been described in various papers submitted to conference (see submitted papers). Vine-growers have also been directly engaged as part of the RAPID project. In particular, we have developed a working relationship with E&J Gallo winery, whereby we have briefed the company about our efforts and findings, and we have obtained access to their ranches for data collection, as well as historic data they formerly gathered. Finally, a presentation about the RAPID project was given at the yearly Merced Ag Tech Fair (March 2017), an event attended by a diverse audience including farmers, community leaders, entrepreneurs, and the general public. Changes/Problems:
Nothing Reported
What opportunities for training and professional development has the project provided?During the first year, we have involved various undergraduates at UC Merced and UC Berkeley. These students have been involved in activities like data collection and field operations (UC Merced) and the design of learning algorithms (UC Berkeley). Moreover, postodctoral fellows have been working on this project at UC Berkeley. How have the results been disseminated to communities of interest?As reported in the appropriate section, at the end of summer 2017 various papers have been written to describe the results obtained so far. These papers are currently under review for the top robotics conference (IEEE International Conference on Robotics and Automation). What do you plan to do during the next reporting period to accomplish the goals?In the next period, we will scale up the data collection effort with the precise objective of feeding real world data into the algorithms estimating water content (objective 2) and determining how the robot should navigate through the environment (objective 3). Moreover, the robotic device for water emitter adjustments will be tested in the field.
Impacts
What was accomplished under these goals?
The objective of this project is to explore the possibility of automating various aspects of precision irrigation thorough robotics and artificial intelligence. This is particularly important for crops like grapes whose quality depends on accurate irrigation, ideally adjusted on a per-vine basis. In particular, data-driven methods combined with automated remote sensing offer the possibility of creating new models and methods to determine soil water content, and accordingly adjust the irrigation schedule. On the actuation side, robotic systems can be deployed witht the objective of adjusting passive water emitters used to retrofit existing irrgation infrastructure. The eventual objective is to reduce water use while keeping the same product quality. In this first year the team has worked towards all of the objectives stated in the project description. In particular: 1. a robotic system that can autonomously latch to and adjust a variable rate water emitter has been designed and tested. The design includes an actuator retrofit with cameras for visual-servoing and the associated vision-based control algorithm. The system is mounted on a robotic arm and has been for the moment tested in an indoor environment. 2- a convolutional neural network has been designed and trained to learn the mapping between aerial images of vineyards and soil water content. The system has been trained using a custom-developed first-order simulator and postively compare to competing learning algorithms. 3- algorithms to route a single robot through a vineyard to adjust water emitter to a desired level have been designed. These multi-objective optimization algorithms aim at adjusting the largest number of water emitters while not exceeding the robot travel time. 4- extensive data collection has been performed in a vineyard located in Snelling (CA) and managed by E&J Gallo. In particular, on a weekly basis a fixed wing drone has been flown over the ranch collecting multipspectral cameras. At the same time, a team of students on the ground has collected data about soil water content using a manual probe. Finally, an autonomous robots has been used to test autonomous navigation algorithms through the vineyward.
Publications
- Type:
Conference Papers and Presentations
Status:
Submitted
Year Published:
2018
Citation:
R. Berenstein, R. Fox, S. McKinley, S. Carpin, K. Goldberg. � Robustly Adjusting Indoor Drip Irrigation Emitters with the Toyota HSR Robot, �ICRA 2018
- Type:
Conference Papers and Presentations
Status:
Submitted
Year Published:
2018
Citation:
D. Tseng, D. Wang, C. Chen, L. Miller, J. Viers, S. Vougioukas, S. Carpin, J. Aparicio Ojea, K. Goldberg. Learning to Infer Local Soil Moisture Conditions from Aerial Agricultural Images for Automating Precision Irrigation, ICRA 2018
- Type:
Conference Papers and Presentations
Status:
Submitted
Year Published:
2018
Citation:
T. Thayer, S. Vougioukas, K. Goldberg, S. Carpin� Routing Algorithms for Robot Assisted Precision Irrigation, ICRA 2018
|
|