Vineyard Groundcover Biodiversity: Using Deep Learning to Differentiate Cover Crop Communities from Aerial RGB Imagery

1. Introduction

First introduced in the late 1980s, “biodiversity” encompasses the diversity of ecosystems, species, and genes in space and time, as well as the interactions among these organizational levels [1,2]. The different levels (genes, species, ecosystems) and dimensions (elements, structures, and processes) of biodiversity directly and indirectly contribute to ecosystem services. Ecosystem services referrer to the value (monetary or otherwise) of ecosystems; they provide, humanity with goods and services for well-being and contribute to agricultural development [3], Contrasting with the loss of habit and biodiversity caused by agricultural intensification [4,5]. In viticulture, the loss of biodiversity can lead to a decrease in both the quality and quantity of grape production [6]. In this context, the viticultural sector should implement vineyard management strategies aimed at conserving biodiversity and promoting sustainable agricultural practices. Some viticultural practices, such as presence of spontaneous vegetation or cover crops between vine rows can increase populations of beneficial insects [7] and provide ecosystem services, including reduced soil erosion, leaching of pesticides, and improved soil structure and biological activity [8,9]. Cover crops can be either sown or spontaneous: according to Vandermeer and Perfecto’s definition, sown crops contribute to planned biodiversity as they are directly introduced by the farmer, while both types support associated biodiversity [10].

Remote sensing, using both unmanned aerial vehicles (UAVs) and satellite sensors, is increasingly prevalent in ecological studies. Its applications range widely, encompassing biodiversity monitoring, assessment of biodiversity-related properties, and employing diverse data, methods, and study areas [11]. Remote sensing also contributes to the quantification and mapping of ecosystem services [12,13], commonly involving the classification of herbaceous plants using RGB [14] and hyperspectral imagery [15,16]. Machine learning (ML), a subset of artificial intelligence (AI), employs algorithms to learn from data and create predictive models [17]. Deep learning (DL), a specialized type of ML, excels at extracting complex features from large datasets through multiple layers of processing [18]. Recent advances in DL, such as those demonstrated by Zhang et al. (2016), show considerable promise for classifying and segmenting landscape features from remote sensing data.

Viticulture increasingly utilizes remote sensing (e.g., UAVs, and satellites such as Sentinel-2 [19]) and AI for vineyard monitoring. These technologies aid in assessing vine health by detecting canopy variations [20] and identifying diseases like Esca [21,22] and flavescence dorée [23]. Applications also include monitoring vine vigor, yield [24], and soil electrical conductivity [25] or soil electrical conductivity [26]. However, the application of remote sensing and AI to characterize ground cover biodiversity in vineyards remains relatively unexplored.

The U-Net [27] is a DL architecture designed for pixel-wise image segmentation. This study employs U-Net for semantic segmentation of UAV RGB imagery. U-Net’s symmetrical encoder-decoder structure, incorporating skip connections, preserves spatial information, enabling the capure of both local and global features, ideal for object detection and boundary delineation. When implementing DL semantic segmentation models, two approaches could be taken, using or not a backbone architecture. The use of a backbone architecture leverage pretrained networks derived from established architectures. The aim of this paper is to develop a DL model that would categorize the pixels into nine classes: vine, soil, and seven cover crop communities. This classification provides a tool for mapping cover crops communities composition of vineyard laying the groundwork for future studies on how cover crops may contribute to the ecological balance and sustainability of vineyard ecosystems through ecosystem services provision. With this aim the discrimination among the main cover crop communities is crucial due to the differing phenology and morphology and different contributions to ecosystem services with which these groups are associated. Future researches are needed to encompass a broader range of vineyard types, soil conditions, and geographic locations and to replicate the use of the DL model on images with a much lower resolution, allowing a considerable reduction of costs and time of imaging acquisition by UAV.

2. Materials and Methods

2.1. Study Site

This study was conducted in a 2-hectare vineyard belonging to Azienda Agricola Ricci Curbastro in Capriolo, Brescia, Italy (Figure 1). The vineyard is situated in the Franciacorta wine region, known for its high-quality sparkling wine, scenic landscapes, and rich history.

2.2. Cover Crop Communities Classification

In this study, in addition to classifying the land cover of vine and soil, we distinguished seven cover crop communities: graminoids, legumes (Fabaceae), mustards (Brassicaceae), composites (Asteraceae), Polygonaceae, Plantaginaceae, and other forbs. These cover crop communities are distinguished by their phenological and morphological differences and play a crucial role in providing various ecosystem services due to their distinct characteristics. Graminoids [27] are represented by Poaceae (grasses in the strict sense), Cyperaceae (sedges), and Juncaceae (rushes), but, are mainly characterized by narrow leaves and fasciculate root systems play an important role in combating soil erosion and weed competition [28]. Legumes (Fabaceae) are widely recognized for the numerous important functions they provide, including nitrogen fixation and the enhancement of soil health and biological fertility [29,30]. The existing literature highlights the biofumigant effects of mustards (Brassicaceae), which demonstrate the suppression of soil-borne pathogens in vineyards and nurseries [31]. Composites (Asteraceae) play an important ecological role by supporting natural enemies [32]. Among the Polygonaceae, one of the main species is buckwheat, which can suppress weeds due to its rapid growth and allelopathic effects, while also hosting many arthropods that contribute to pest control [33]. Mixtures of cover crops containing Plantaginaceae show significant suppression of weeds [34]. Generally, forbs, with their tap root systems, contribute to soil structure improvement and, compared to graminoids, also differ in terms of impact on the dynamics of organic carbon in the soil [35].

2.3. Image Acquisition

A flight was conducted on September 21, 2023, using an M300 RTK UAV with a flight autonomy of up to 55 minutes. The drone was autonomously operated using Litchi software (version 4.25.0, released July 4, 2022, by VC Technology Ltd., London), which enabled a comprehensive photogrammetry mission. Flight parameters, including an 8-meter altitude above ground level, four ground control points (GCPs) with precise coordinates, and image resolution, were configured to ensure complete coverage of the study area. A total of 24 images were acquired.

2.4. Data Preprocessing for Cover Groups Segmentation

The study utilized 24 RGB images (8192 x 5460 pixels), each representing 40 square meters, for a total analyzed area of 960 square meters. While DL offers advantages over traditional ML for plant species mapping by handling complex data, the large image size presents computational challenges. To address this, the images underwent preprocessing (Figure 2) as follows:

• A plant expert manually annotated masks for each training image using Roboflow software.
• The high-resolution images and their corresponding masks were partitioned into 256 x 256-pixel patches for training a U-Net segmentation model. This facilitated efficient image segmentation and feature identification. The trained model was then saved.
• The saved model was used for the prediction and mapping of the full-size images.

Different types of ground cover (cover crops, vines, and soil) were manually labeled by an experienced botanist with in-depth knowledge of vineyard ecosystems. The expert classified each polygon in the mask image, which had dimensions of 8192 x 5460 pixels, assigning labels for the established reference categories when at least 60% of the category was recognized based on vegetative morphological characteristics [36]. Subsequently, the image was divided into patches of 256 x 256 pixels, and the classification was verified by the same expert to ensure accuracy and consistency. This study addressed the challenges posed by the large size (8192 x 5460 pixels) of the original images, which caused high memory requirements and slow training times in DL models. A Python script was developed to efficiently process these images. The script iterated through image and mask file directories, resizing each image and mask to the nearest size divisible by 256 pixels (Figure 3). The patchify library (TensorFlow) then segmented the resized images and masks into non-overlapping 256 x 256-pixel patches. These patches were saved individually, maintaining their association with corresponding masks. Finally, the splitfolders library (TensorFlow) split the patches into training (75%) and validation (25%) datasets, resulting in 11,700 training and 3,900 validation images. This preprocessing enabled efficient model training and evaluation.

2.5. Semantic Segmentation Model

A UNet semantic segmentation model [37], implemented in Keras (Python), was designed for pixel-wise classification of UAV RGB imagery. This architecture, illustrated in Figure 4, comprises encoder and decoder pathways with skip connections to preserve spatial information and effectively capture both local and global features. The model outputs a segmented mask classifying each pixel into one of nine classes: seven cover crop communities (graminoids, legumes, mustards, composites, Polygonaceae, Plantaginaceae, and other forbs), vine, and soil. The training involved minimizing a loss function (measuring the discrepancy between predicted and ground truth masks), with performance evaluated using Intersection over Union (IoU) [38], accuracy, and Mean IoU. We compared semantic segmentation with and without backbone architectures. The “no backbone” approach trained the UNet from scratch, relying solely on its inherent architecture for feature extraction. This was contrasted against models incorporating four popular backbone architectures: ResNet [39], a deep convolutional network utilizing residual connections to mitigate vanishing gradients; InceptionV3 [40], known for its computational efficiency and multi-scale feature extraction via inception modules; EfficientNet [41], distinguished by its compound scaling method for optimal resource utilization; and DenseNet [42], characterized by dense connections between layers promoting feature reuse and efficient gradient propagation.

3. Results

Our UNet semantic segmentation model was trained on a dataset comprising 15,600 image patches. These patches were derived from 24 high-resolution RGB images (8192x5460 pixels) by dividing them into smaller 256x256 pixel sections. To ensure rigorous evaluation, the dataset was divided into a training set (70%), a validation set (15%), and a test set (15%).

Each image is annotated with one of nine classes: seven cover crop communities, vine, and soil. Initially, the images and their corresponding masks are loaded from the specified directories and pre-processed accordingly. The masks are encoded using label encoding and then split into training and testing datasets. Next, the model is configured with specific parameters, including the DensNet121, ResNet34, InceptionV3, and EfficientNet backbone architecture, and a combination of Dice loss and Categorical Focal loss as the optimization objective. The model is trained over 100 epochs, and the training progress is visualized using plots for loss and IOU score. After training, the model is saved for future use. Performance metrics such as accuracy, precision, recall, F1 score, Jaccard score, and Mean IoU are computed using the trained model on the test dataset. Finally, the results are printed to evaluate the segmentation model’s performance.

3.1. Class Imbalance and Data Augmentation

To address the class imbalance evident in Table 1, where certain classes (e.g., mustards, Polygonaceae, and Plantaginaceae) have significantly lower coverage areas than others, we implemented a class-balanced data augmentation strategy. This involved selectively augmenting the training data by creating new image patches that focused on the under-represented classes. Specifically, we modified the existing masks to temporarily ignore highly represented classes, effectively creating new masks where the under-represented classes became the dominant features. This process increased the number of training samples for the under-represented classes without altering the original dataset. New image patches were then generated from these modified masks, thus enriching the training data with a more balanced representation of all classes. This targeted approach ensured that the model received sufficient training examples for even the least prevalent classes, improving its ability to accurately segment these features.

3.2. Overfitting Prevention Strategies

To mitigate overfitting, we implemented early stopping. Training was halted when the validation loss stopped decreasing for a specified number of epochs (e.g., 10 epochs), ensuring that training continued only as long as the model was improving its performance on unseen data. Additionally, batch normalization was employed to stabilize and accelerate training, and class weighting in the loss function helped manage class imbalances. These strategies collectively contributed to preventing overfitting and enhancing the model’s generalization ability.

3.3. Model Performance

The comparison of semantic segmentation models with different backbone architectures reveals nuanced differences in performance metrics (Table 2). Across all models, including ResNet34, EfficientNet, InceptionV3, DenseNet, and a model without a specified backbone, there is a notable consistency in accuracy, precision, recall, and F1 score, with variations typically within a range of 1-2 percentage points. However, when assessing metrics more tailored to semantic segmentation tasks, such as mean IOU and Jaccard score, subtle disparities emerge. EfficientNet and DenseNet exhibit slightly higher mean IOU and Jaccard scores compared to ResNet34 and InceptionV3, highlighting their marginally superior ability to accurately segment objects in images. For instance, EfficientNet achieves an accuracy of 85.4%, while DenseNet reaches 83.6%, both with corresponding mean IOU scores of 59.8% and 52.1%, respectively. These results underscore the importance of selecting an appropriate backbone architecture, as models with dedicated architectures designed for image segmentation tasks demonstrate enhanced performance, particularly in terms of mean IOU and Jaccard score, compared to models without a specified backbone.

After loading the trained model, we generate a batch of test images and masks using the validation data generator. Next, we compute predictions for the test images using the loaded model and convert the predicted masks from categorical to integer format for visualization and IoU calculation. Finally, we visualize a randomly selected test image along with its corresponding ground truth mask and predicted mask for a qualitative assessment of the model’s performance (Figure 5).

Finally, the trained model was loaded to apply it to a large image, where it was segmented into patches of appropriate size for processing. The prediction process was conducted on each patch, and the resulting segmented patches were then stitched together to reconstruct the predicted mask for the entire large image, facilitating the application of the model to images beyond the validation dataset (see Figure 6).

4. Discussion

Our study aims to establish a method for identifying vineyard inter-row groundcover, with the ultimate goal of creating a comprehensive groundcover map and supporting decision-making for soil and cover crops management in viticulture. In this work, the potential of DL techniques applied to RGB images was evaluated in a vineyard with an extension of about 2 hectares located in the Franciacorta wine-growing area. The main focus of the study was to distinguish between nine classes of groundcover. Beyond the two classes vine and soil, other seven different groups of cover crops communities were distinguished: graminoids, composite, mustards, legumes, Polygonaceae, Plantaginaceae, and other forbs. These groups, differentiated by phenology and morphology, play a crucial role in providing various ecosystem services. specifically, graminoids, thanks to their fasciculate root system, contribute to the improvement of the structure of the upper soil layers and to the limitation of soil erosion. Whereas the different forb groups, contribute to a variety of services to the vineyard, such as improving soil structure at a greater depth, nitrogen fixation, among others [28,35].

Using a total of 24 RGB images, taken with a done at 8 m height, five different artificial intelligence models based on the UNet architecture [37] were trained and tested to assess their accuracy in predicting the observed soil cover. By employing a UNet architecture, which excels in pixel-wise classification tasks through its symmetrical encoder-decoder structure and skip connections, the model effectively segments UAV RGB images into predefined classes such as vine, soil, and various vegetation types. The integration of advanced DL techniques, such as semantic segmentation with backbones, offers a notable improvement in accuracy over methods without backbones, as evidenced by performance metrics. The use of backbones like EfficientNet [41], ResNet [39], InceptionV3 [40] and DenseNet [42] provided a robust foundation for feature extraction, enhancing the model’s ability to capture hierarchical features and spatial relationships within the images. Even though all the architectures analyzed showed high performance, exceeding an accuracy of 70%, the UNet model with an EfficientNet backbone stood out above the rest, achieving an accuracy of 85.4%, with a precision of 84.97 %. Unlike generalized datasets, ours is tailored to the unique ecological conditions and management practices of vineyards, making direct comparisons with other segmentation benchmarks difficult. To assess our UNet model, we therefore had compared it to relevant studies in agricultural image segmentation using aerial imagery. While many studies, like those by Zuo and Li [43] on weed segmentation in corn fields and Shai et al. [44] on weed detection using UAV images, focus on different crop types and scales, our dataset’s emphasis on vineyard cover crops necessitates a more granular analysis. Our UNet architecture aligns with common practices in image segmentation, as exemplified by [45,46].

The results of this study highlight the potential of DL models, particularly those utilizing advanced backbone architectures, to enhance the precision and efficiency of groundcover classification in vineyards. This advancement in technology provides valuable insights into the spatial distribution and ecological roles of different groundcover types, which can significantly aid vineyard management practices. Specifically, this study represents a first step in applying DL to the classification of cover crop communities in vineyard agroecosystems. The results demonstrate that innovative application of these technologies can address complex, long-standing challenges such as monitoring vineyard biodiversity. Moreover, results obtained in the present study, provides the groundwork for future studies on how cover crops may contribute to the ecological balance and sustainability of vineyard ecosystems through ecosystem services provision. However, some limitation has to be underlined. A first key limitation is the potential for bias arising from class imbalance in the dataset. While class-balancing techniques (targeted data augmentation and class weighting) were employed, under-represented classes may still bias predictions, particularly with unseen data. Future research should address this by expanding the dataset to encompass a broader range of vineyard types, soil conditions, and geographic locations. In terms of outputs obtained, although the DL model developed is able to automatically discriminated among nine classes of groundcover, it needs to be evolved in order to generate a real operational tool. Specifically, the results obtained with the DL model on high resolution images, must be replicated on images with a much lower resolution, allowing the automatic identification on a much larger surface (hectares). This represents an interesting perspective of future development of the research, allowing a considerable reduction of costs and time of imaging acquisition by UAV.

5. Conclusions

This study highlights the potential of using remote sensing and artificial intelligence for mapping of groundcovers within vineyards inter-rows, in order to monitor the cover crops communities in vineyard agroecosystems. The comparative analysis conducted in this study shows that the Unet model, with the efficientNET backbone architecture outruns the other architectures highlighting its utility for groundcovers within vineyards inter-rows classification. This study validates the capabilities of the described methodology to create a DL model capable of classify cover crops communities, with the aim of monitoring biodiversity in vineyards, in order to be used in decision-making for soil and cover crops management. Further research are needed to allow the model to monitor larger surface with time and cost advantages and expanding the dataset to encompass a broader range of vineyard types, soil conditions, and geographic locations.