Progress 08/15/16 to 02/14/17
Outputs
Target Audience:Azavea has developed a network of partner organizations who advise on the development of our community forestry products, ensuring that they meet the combined needs of our designated target audiences. Through consultation with our partners, combined with extensive e-mail and phone outreach undertaken during the course of the grant timeframe, Azavea has both confirmed our original target audiences of municipal governments and citizen science organizations for OpenTreeID and identified two additional potential audiences. First, middle schools, high schools, colleges, and universities are potentially interested in making tree species identification a regular part of their curriculum. There are also a growing number of citizen science nonprofits that support environmental learning activities outside the traditional classroom. OpenTreeID offers an interactive e-learning experience that will meet a broad range of educational needs. Second, based on our success in applying computer vision to tree leaves for species identification, we also explored using machine learning and computer vision for classifying land use and tree canopy from aerial imagery. Many municipal governments, regional planning commissions, and sustainability groups use canopy coverage as a method for understanding the urban forest and setting tree planting goals, so adding these capabilities to OpenTreeID offers additional value to this audience. Our initial research found that machine learning techniques applied to aerial imagery efficiently extrapolate land use classifications, canopy coverage percentages, and potentially species distributions. The limited timeframe for our Phase I project, combined with leaf-off conditions when the prototype was completed, meant that hands-on testing was necessarily limited. Two urban forestry subject matter experts represented the needs of our target audiences. Their feedback and testing of the completed application was informative and resulted in a better overall user experience. The resulting prototype is more effective for both urban forestry organizations and individual users because of their contributions. Further, our extension of the effort to aerial imagery enabled us to work with historic imagery and perform comparison tests for a broader range of time periods. Changes/Problems:Much of Azavea's Phase I effort focused on researching and selecting a matching methodology that would combine with questions posed to the user to create a higher match-accuracy rate than existing tree species identification systems. In so doing, we identified and addressed three unexpected challenges to our original approach and forged a new path for OpenTreeID that will significantly expand its commercialization potential. Challenge 1: Switching Matching Methodologies One of the most critical technical challenges for this research was determining the optimal combination of statistical approach and morphological characteristics needed to achieve a high match-accuracy rate. We reviewed two main approaches with several specific variations within each approach. The first was to improve the segmentation algorithm, which enables the software to recognize what is a leaf and what is the background image. The second was to implement a convolutional neural network, which learns the expected outcomes for a proposed question based on the answers fed into the system through a series of training images. While we were originally planning to improve the segmentation algorithm, our research suggested that implementing a convolutional neural network would increase the likelihood of more accurate matches, and our testing confirmed extremely accurate results. That said, the decision to switch from a segmentation process to a convolutional neural network did cause a slight delay in beginning the initial development of the project and required us to gain more experience in implementing neural networks. However, that experience is beneficial to both expanding OpenTreeID in a future phase of the project and applying machine learning to other projects at Azavea. Challenge 2: Developing Human-in-the-Loop Questions Creating the list of questions to pose to users was more challenging than we expected. Many of the questions we considered did not have clear answers that non-arborists could distinguish by looking at the tree, or depended on tree features, such as flowers and fruit, which are not visible year-round. While the four questions we implemented resulted in more accurate results, some questions regarding the shape of the leaf or the description of the leaflets reflect characteristics that the neural network should identify on its own. Ideally, the probabilistic questions would relate to information such as the shape of the tree, bark color, fruit, or other species characteristics not connected to the leaf, because the neural network has already identified those features. Challenge 3: Testing and Expansion ?Azavea completed the OpenTreeID prototype application in January 2017, which made it very challenging to conduct any field testing in Philadelphia due to leaf-off conditions. While we were able to find a few leaves on trees near our homes and office to use for testing, we were not able to extensively use the application out in the field. For that reason, we completed the majority of testing using leaf images found online. The success metric we articulated in our Phase I proposal was to exceed the 68% first-place match-accuracy rate achieved with Leafsnap. While we expected the convolutional neural network to result in accurate species identification, our match-accuracy rates averaged 96% and generally exceeded 80%, while reaching 99% in many case. This success exceeded our most ambitious expectations. Initially, we were concerned that the high level of accuracy indicated an error in our programming and the tests did not reflect accurate results. After reviewing our work in training the neural network and the image databases, we confirmed the accuracy of the test results. Even more exciting from a commercialization perspective was the realization that we could apply similar machine learning techniques to other underserved needs in community forestry, including extrapolating tree canopy assessment, tree genus and species distribution, and land use classification data from publicly available satellite imagery. We are only a few years away from being able to capture remotely sensed imagery for any location on the planet on a daily or even more frequent basis. As the data volume goes up and the cost of acquiring data is driven down, the chief value is in making the imagery useful by either processing it for visualization or transforming it into new information products. Extending OpenTreeID to support land use classification from remotely sensed data provides a substantial head start on meeting these challenges and enables global application to a growing array of industries that extend far beyond community forestry. What opportunities for training and professional development has the project provided?The Urban Ecosystems team completed primary development on OpenTreeID, but there were three software engineers from other teams that had considerably more experience with convolutional neural networks (CNN) and machine learning. Azavea was able to coordinate multiple project deadlines so that these engineers could contribute to research and decision-making tasks on OpenTreeID. This saved us from completing extensive and potentially costly tasks implementing an alternate species identification methodology that was unlikely to result in more accurate matches. At the same time, it provided valuable cross-team collaboration and knowledge-sharing that will benefit other projects going forward. Finally, it enabled the software developers and project manager on the Urban Ecosystems team to gain valuable experience and new skills with respect to machine learning and computer vision. How have the results been disseminated to communities of interest?Azavea reviewed the prototype application with our technical and urban forestry consultants on our advisory board and made changes based on their feedback. While the prototype will not be released to the general public, we have demonstrated the application to several colleagues working in the urban forestry field and received positive reviews. Our research into using machine learning on aerial imagery also demonstrated successful results. We submitted these results to a global competition organized by the International Society for Photogrammetry and Remote Sensing (ISPRS) and are currently generating the second most accurate results (with a group from the Chinese Academy of Sciences generating the most accurate results at 91.1% accurate). A blog post has been developed regarding this portion of the project (https://www.azavea.com/blog/2017/05/30/deep-learning-on-aerial-imagery/). The software source code is currently hosted on GitHub and will be featured in a forthcoming edition of Azavea's company newsletter. What do you plan to do during the next reporting period to accomplish the goals?
Nothing Reported
Impacts
What was accomplished under these goals?
Azavea's goal for OpenTreeID is to significantly lower the barriers to entry for small to mid-sized communities that cannot currently afford state-of-the-art data collection and analytical tools for tree canopy assessment, land use classification, and species identification. The consulting costs to gather and interpret this type of data currently range from tens of thousands to many hundreds of thousands of dollars for a single community. The ability to perform such services in-house can be similarly constrained by the need for advanced analytical skills, specialized software, sophisticated classification schemes, and significant investments in staff time. OpenTreeID offers automated tree species identification, tree canopy assessment, and land use classification capabilities in an easy-to-use, subscription-based solution specifically designed for organizations with limited technical or financial resources. To our knowledge, no other community forestry solution currently provides this unique combination of capabilities. The original intent of our Phase I project was to improve upon existing tree identification systems, such as Leafsnap, by combining computer vision and machine learning techniques with human decision-making. This, in turn, would increase the overall accuracy rate of crowdsourced tree data that is critical to contemporary community forestry. To that end, we identified and addressed four key technology objectives: 1.) Compile a Training Database Working with two community forestry advisors, we selected twenty-five species commonly found on urban streets and parks in the Northeast United States. We also created a database of images from the existing publicly available databases to train the automated species identification system. We later expanded this species list and the corresponding images based on successful testing of our twenty-five initial species. 2.) Determine a Matching Methodology Azavea chose a matching methodology combined with questions posed to the user to create a higher match-accuracy rate. After assessing two potential approaches, we implemented a convolutional neural network, an image classification algorithm that learns the expected outcomes for a proposed problem (such as selecting the correct tree species) based on comparing an input image with a curated set of training images. We also developed a series of leaf-related questions to be posed to the user in order to narrow down the potential results identified by the software. 3. Develop Identification Algorithms Azavea consulted with a machine learning advisor to select and update a convolutional neural network model that automates the species identification process. We then trained the network with photos of leaves, including images with plain backgrounds and images with noisy backgrounds such as sidewalks and other leaves, to enable it to distinguish one species from another. 4. Design and Build a Local Prototype The resulting prototype uses Keras, a high-level neural networks library; TensorFlow, an open source machine learning library originally from Google; and Amazon Web Services, a cloud computation infrastructure. It is accessible as a web application optimized for the Chrome browser on mobile devices. In initial testing, our prototype has achieved accuracy results significantly exceeded our metrics for success. Users open the OpenTreeID web application, hold a leaf in front of the camera on their smartphone, and quickly receive suggestions on the proper identification of the tree's species. OpenTreeID's first-place match-accuracy rate is between 88 and 95%, which is a significant improvement over Leafsnap's 68% match-accuracy rate with the same species. Based on successful completion of the four technical objectives outlined above and with time remaining in the grant period, we expanded our convolutional neural network implementation to explore how the same class of machine learning techniques could be applied to analyzing aerial and satellite imagery in order to distinguish between different types of land cover in aerial imagery on a pixel-by-pixel basis. These are important components of community forestry plans and canopy coverage goals that serve as a benchmark for future tree planting and maintenance goals. Using semantic segmentation, a machine learning process that makes a classification determination for each pixel in an aerial image, Azavea has been able to automate the application of land use classes that include trees, impervious surfaces, buildings, low vegetation, cars, and clutter/background. Further, we have identified multiple potential methods for extrapolating tree genus and species distribution from aerial imagery. Use cases in agriculture, international development, urban and regional planning, environmental management, public utilities, and sustainable transportation will range from vegetation analyses to watershed management and from targeting locations for green infrastructure to studying forest fragmentation over time.
Publications
- Type:
Other
Status:
Published
Year Published:
2017
Citation:
https://www.azavea.com/blog/2017/05/30/deep-learning-on-aerial-imagery/
|
Progress 08/15/16 to 02/14/17
Outputs
Target Audience:Our target audiences for this project are the municipal governments, nonprofits, and citizen foresters that contribute to the health and maintenance of the community forest. While we did not work directly with any of these audiences during this reporting period, we made significant progress toward developing an application that will meet their needs. Our urban forestry consultants, provided valuable expertise and thoughtfully considered which species to represent in thePhase I prototypebased on the various reasonsour target audiencesmay be using the software. They raised questions we had not previously considered regarding invasive species, pests and diseases, and the process of posing questions to users. The resulting software is much more useful to both urban forestry organizations and individual users because of their contributions to the project. Changes/Problems:Azavea encountered three unexpected challenges thatneeded to beaddressed during the first three months of this project. First, we hadother urban forestry work that temporarily shifted our focus. OpenTreeID may eventually be included as an addition to OpenTreeMap, our software for collaborative urban tree inventory. Over forty cities, non-profit groups, and businesses now subscribe to the OpenTreeMap Software-as-a-Service (SaaS) system, and several of those clients have contracted with us to add custom features to the system. Our success with OpenTreeMap has provided us with both revenue and a wealth of user feedback, but it has required us to periodically focus on client-funded features that had not previously been included in the development schedule. We currently expect that these additional development obligations will be reduced over the next few months. Second, implementing a convolutional neural network is a different approach than we had originally planned. Our research suggests that this approach will improve the likelihood of more accurate matches. This conclusion is based on our research into neural networks as well as a thorough literature review showing demonstrated success with using CNNs on other plant related machine learning projects. This shift in methodology may not result in the successes we expect, however, and, if this turns out to be the case, we will then need to return to our methodology research to test a different approach. Third, while we had originally planned to design the system to use GeoTrellis, Azavea's high performance geoprocessing engine and toolkit, additional research found that Amazon Web Services provides GPU-based instances that will support increasing the speed of the convolutional neural networks significantly compared to traditional Central Processing Units (CPUs). This will provide sufficient processing capabilities to support the project. What opportunities for training and professional development has the project provided?Through its focus on machine learning, this project has provided opportunities for professional development and knowledge sharing across multiple software teams at Azavea. When we originally proposed the OpenTreeID system, we had primarily considered implementing a methodology that improved upon the segmentation algorithm used by an existing system. Upon further research, however, we decided on implementing a convolutional neural network to support recognizing the leaf without a separate segmentation step. That decision was made after extensive research and collaboration with software engineers on multiple teams at Azavea. Roundtable discussions between these engineers and the Urban Ecosystems team that will be primarily responsible for developing OpenTreeID saved us from completing extensive and potentially costly tasks or implementing a methodology that was unlikely to result in more accurate matches. In so doing, this knowledge exchange benefitted both the Urban Ecosystems team and the project as a whole. How have the results been disseminated to communities of interest?The results of this project to date have been shared with members of the Azavea staff as well as our urban forestry consultantsand our technical consultant. We anticipate broader dissemination during the last half of the project, once the prototype is operational. What do you plan to do during the next reporting period to accomplish the goals?In the second half of the grant period, Azavea will test ourspecies identification methodologiesagainst sample data and work with our urban forestry and computer vision consultants to create a user interface for the OpenTreeID web application. Creating this interface will enable us to conduct some basic user testing and gain valuable insight into user needs to inform our commercialization plans that currently focus on non-profit groups and government agencies.
Impacts
What was accomplished under these goals?
The first half ofAzavea's Phase I project has been focused on identifying a matching methodology to improve upon existing leaf identification systemsand building a prototype application to test its utility. OpenTreeIDcombines computer vision techniques with human decision-making to more efficiently and accurately identify a tree's species based on the shape of its leaves. Users will open the OpenTreeID web application, hold a leaf in front of the camera on their smartphone, and quickly receive suggestions on the proper identification of the tree's species. By using a mobile phone, which many individuals already carry with them regularly, OpenTreeID eliminates the need for a bulky tree identification booklet and enables people to learn and contribute useful dataabout the urban forest at any time. Azavea is developing OpenTreeID by way of four technical objectives and a series of related tasks. Our first objective was to compile a training database of leaf images that will be used to train the automated species identification system. Working with two urban forestry subject matter experts,weselected twenty-five species for this initial phase of work that are common to our pilot location of the City of Philadelphia. Weresearched possible training image datasets from publicly available projects including Leafsnap, PlantCLEF at the Life CLEF Lab, and the Leaf Classification Project using images from the Royal Botanic Gardens at Kew, United Kingdom. Only the Leafsnap database included images for all the species selected by the urban forestry consultants and featured a variety of images captured in the field, which assists with training the system. We therefore plan to use Leafsnap imagery for our training database. Our second objective focuses on researching and selecting a matching methodology that will combine with questions posed to the user to create a higher match-accuracy rate. After an extensive review of various machine learning methodologies, we chose to implement a convolutional neural network based on previous research showing success with such networks for plant identification. Because It is not possible for computer vision and machine learning to resolve all identifications accurately 100% of the time, we are also creating a database of species characteristics (bark texture, leaf size, etc.) that we will use to ask users questions about the trees. We expect that adding this "human-in-the-loop" element to distinguish between two possible species suggested by the neural network will greatly increase the accuracy of results. Our third objective is focused on developing algorithms that will both automate the identification process and enable it to be refined and enhanced through machine learning techniques. In coordination with our machine learning subject matter expert, Azavea staff reviewed the OpenCV (Open Source Computer Vision Library) computer vision and machine learning software library for potential algorithms that could be adapted to this project. After much discussion, we chose to proceed with implementing a convolutional neural network approach, which is a software program that learns the expected outcomes for a proposed question based on the answers fed into the system through a series of training images or questions. Training a neural network with photos of leaves in multiple orientations, including both plain backgrounds and consistent but noisy backgrounds, like sidewalks, will result in a network that can accurately identify the leaf species without an additional segmentation step to separate the leaf from the background. Should the convolutional neural network return results that are not as accurate as we would like, we may implement a segmentation step later on in order to improve match accuracy. Our fourth and final objective is to design and build a local prototype as a web application that is viewable on mobile and tablet devices.We expect to hold our design charrette in the coming weeks with our urban forestry consultants and the User Experience (UX) Designers at Azavea. The charrette will include developing personas to represent the expected users of the system and developing rough sketches based on their goals for the application.
Publications
|