Vision Transformer Based Unhealthy Tree Crown Detection and Evaluation of Annotation Uncertainty

1. Introduction

Forest ecosystems provide essential services, such as carbon sequestration and biodiversity conservation, yet human activities and environmental stressors, such as climate change and insect infestations increasingly threaten them [1,2,3,4]. These stressors contribute to declining tree health, resulting in biodiversity loss [5], altered hydrological and carbon cycles [6], and increased risks to human safety [7], particularly in vulnerable areas, such as overhead electric utility corridors. The health of trees in utility corridors is particularly concerning, as compromised tree conditions can result in infrastructure damage and public safety hazards. Furthermore, forest disturbances are expected to become more frequent and severe due to the synergistic effects of human activities and climate change, particularly in areas like the Northeastern United States, where forest cover constitutes a significant portion of the landscape [8].

Managing degraded and dying forests at various spatial levels, from individual trees in urban areas to larger stands on a landscape scale is challenging. Effective management of trees is reliant upon accurate and up-to-date information. To maintain forest dynamics and promote sustainable forest management practices, monitoring forest health and timely identifying unhealthy trees is essential. Traditional inventory methods are often time-consuming and costly, but advancements in remote sensing (RS) technology, digital image processing, and machine learning offer promising alternatives for assessing unhealthy tree crowns (UTCs) [9,10,11].

Different automated image analysis methods in RS have advanced significantly, mainly with deep learning (DL) techniques, such as convolutional neural networks (CNNs) for object segmentation [12], classification [13], and detection [14]. CNN offers a robust and more accurate approach to classification than traditional approaches [15,16] by directly identifying objects in training data and learning hierarchical combinations of image features focusing on object-level representations. Also, CNNs are re-trainable for incorporating unique dataset characteristics [17], thus excelling in segmentation and detection tasks, such as individual tree crown detection [18,19], plant health monitoring [9,20,21,22,23], forest mapping and monitoring [24,25,26]. [27] reviewed the application of CNNs in forest remote sensing and noted that CNNs outperform shallow machine learning techniques, such as random forest or support vector machine algorithms in a wide range of forest applications.

CNN models, such as Faster R-CNN [28], FCN [29], Mask R-CNN [30], U-Net [31], DeepLab v3+ [32], and YOLO [33], have been widely used for segmentation tasks. For instance, [34] utilized Unmanned Aerial Vehicle (UAV)-based RGB images to semantically segment individual species in an urban environment. They compared the performance of five different CNNs and achieved accuracy ranging from 88.9% to 96.7% across various architectures in identifying endangered single species. Also, CNNs have proven effective in tasks, such as tree counting and identification, using high-resolution remote sensing data outperforming traditional methods [9,15,35]. [19] designed and implemented a CNN-based deep learning framework to detect oil palm trees in high-resolution images with an accuracy higher than 96% while counting the individual treetops. [36] were capable of mapping more than 1.8 billion trees in the Sahara and Sahel region using more than 11 thousand satellite scenes. It shows the potentiality of CNNs in pattern recognition for vegetation analysis from RS data. Also, DL-based techniques are becoming prevalent in leveraging 3D-LiDAR datasets [37,38,39,40] for accurate segmentation tasks. LiDAR-derived canopy height models, and other information coupled with multispectral images and/or hyperspectral images, are gaining popularity due to their adaptability, and showing flexibility in both spatial and spectral resolution [17]. However, tree detection and segmentation tasks using CNNs remain a challenge due to the lack of training samples, complex geometries, target-background imbalance, imaging conditions, and sensor characteristics. Moreover, accurately identifying targets in complex scenes, such as mixed dense forest vegetation, solely based on local feature information, presents a challenge for CNN-based models.

Vision Transformers (ViTs) [41] is a transformer [42] like architecture introduced in the field of computer vision, specifically designed for the vision task [43] to understand the spatial relationships in an image. Transformers are powerful because they can be trained on massive internet data, and then fine-tuned to transfer their learned abilities to excel at other domains using smaller, task-specific datasets [41]. ViTs have demonstrated significant application in semantic segmentation tasks, often retaining the encoder-decoder architecture from CNN-based approaches. This makes them versatile tools for a wide range of applications, including remote sensing image analysis [44,45,46,47]. For instance, [45], used a ViT for the remote sensing scene classification and achieved above 94% accuracy in different datasets. [48] have reviewed a large body of studies that compared the performance of CNNs with ViTs and concluded that ViTs outperformed CNNs. However, ViTs still have certain disadvantages when compared to CNNs, such as the necessity for extremely large datasets. Self-supervised techniques can mitigate some of these limitations and enhance ViTs further [49]On the other hand, combining CNNS and ViTs for classification purposes can help overcome the limitations of either approach. For instance, [50] combined CNN and transformer for the agricultural crop classification in the time-series satellite images. They first extracted multitemporal features from CNN and then used transformers to learn the land cover patterns.

ViTs offer a promising alternative to CNNs for semantic segmentation tasks, particularly those requiring global context, high-resolution images, or intricate segmentation. SegFormer [51], Swin Transformer [52], Detectron2 with transformer backbone [53], Pyramid Vision Transformer (PVT) [54], and Masked-attention Mask Transformer (Mask2Former) [55] are popular ViTs for semantic segmentation tasks. Such ViTs are gaining popularity in the field of remote sensing for semantic segmentation tasks [56]. The SegFormer algorithm [51], a notable transformer-based segmentation model inspired by ViTs, partitions the input image into smaller patches and feeds each patch as a sequence into the transformer for analysis. Unlike conventional semantic segmentation techniques, the SegFormer model integrates multi-scale features to capture pixel relationships, thereby reducing parameters and computations in the decoder section effectively. In forest remote sensing, a very limited number of studies have used transformer-based semantic segmentation. ViTs are mainly used in land cover classification and change detection purposes [47,57,58,59]. There is a need and an opportunity to explore the performance of ViTs in forest health monitoring tasks.

Forest health monitoring in a dense mixed forest has significant challenges as individual spectral bands may not suffice to detect unhealthy tree canopies. Vegetation indices (VIs), derived from combinations of spectral reflectance values, are proposed as an alternative to individual spectral bands [60]. However, VIs have limitations in interpreting forest canopy information, especially regarding variables, for instance, chlorophyll content and leaf angles [61]. [62] found that the performance comparisons with spectral mixture analysis show VIs' shortcomings in assessing tree canopy cover. Also, several studies claimed that textural information can improve forest health classification [63,64,65]. However, there is still a significant amount of work needed to assess the effectiveness of texture analysis in examining tree decay and mortality [66]. Most DL models applied in satellite and aerial imagery analysis, particularly segmentation, predominantly employ end-to-end architectures that utilize multi- or hyper-spectral bands directly in their processing pipeline. However, challenges arise from the complex, multimodal, geolocated, and multitemporal nature of the data, coupled with limited labeled remote-sensing data for model training purposes. Additionally, the growth of high-dimensional data surpasses improvements in GPU (Graphical Processing units) computing power. To improve the efficacy of DL models in high-dimensional data analysis, it is essential to incorporate domain knowledge from forest science and remote sensing. [67] suggest that the integration of theory-driven insights with data science significantly bolsters model robustness in complex environmental settings. By embedding textural information and VIs as prior knowledge, we aim to refine DL models and enhance their predictive capabilities in forestry applications.

The efficacy of DL models is significantly contingent upon the quality of training samples, where accurate and consistent annotations play a pivotal role in determining model performance. It is imperative to account for annotation uncertainty during model evaluation, as variability in annotations can profoundly affect predictive accuracy and model generalizability. This issue is particularly pronounced in scientific studies that often rely on limited training datasets due to the high costs of data production and the restricted availability of domain experts. Research by [68,69,70] underscores the critical impact of annotator variability, highlighting the necessity for systematic analyses of inter-annotator agreement. By rigorously addressing annotation uncertainty, our study not only strengthens the robustness of model evaluation but also introduces a novel framework that extends beyond conventional metrics, thereby enhancing the reliability and validity of findings in complex environmental contexts. This approach voices the importance of methodological rigor in annotation processes, ultimately contributing to more accurate and explainable DL applications in the field.

In this study, we present a systematic comparative analysis of CNNs and a ViT model to identify unhealthy forest canopies in mixed dense forest areas using publicly available aerial imagery acquired by the National Agricultural Imagery Program (NAIP). We compare the effectiveness of a set of candidate CNNs and ViT for this task, which, to our knowledge, has not been explored before in forest health monitoring in dense mixed forests type. Our specific objectives are to; i) compare the accuracy of the CNN-based models: U-Net, DeepLab v3+, and ViT-based model: SegFormer, ii) evaluate the different models of varying band combination from the 4-band NAIP images that include derived VIs, Principal Components, and image textures, and iii) evaluate manual annotation uncertainty and its impact on evaluation metrics. By examining annotation uncertainty, we aim to shed new light on how inter-annotator variability can impact model performance. Our study seeks to contribute to the development of a cost-effective method for detecting unhealthy trees using high-resolution imagery, which can be valuable for forest management and conservation efforts.

2. Materials and Methods

2.1. Study Area

The study area is in the eastern part of Windham County, Connecticut, USA (Figure 1). Connecticut (CT) is a densely forested state of the USA with a mixed forest vegetation type occupying around 57% percent of land area [8]. The forest type is temperate deciduous, with 58 species of trees in Connecticut [71]. It has a predominant oak-hickory species forest type that covers approximately 70 % of the total forested area, followed by elm, ash, cottonwood, maple, beech, birch, and other hardwoods [71]. These areas have substantial numbers of eastern white pine, eastern hemlock, and other softwood. The forest of CT has been widely infested with invasive insects leading to defoliation and tree mortality. A massive oak tree die-off occurred in eastern CT during 2017-2018. Prolonged drought followed by severe spongy moth (Lymantria dispar) defoliation weakened the trees, making them susceptible to additional stressors like borers and fungus, leading to widespread mortality [72].

2.2. Modeling Framework

The overall methodology is divided into four steps: 1) data retrieval and preprocessing, 2) DL model training and optimization, 3) DL model comparison and accuracy assessment, and 4) annotation uncertainty analysis (Figure 2). NAIP aerial imagery was annotated in the ArcGIS Pro 3.1.2 software, creating polygon labels for the UTCs in shapefile format, which were later turned into a binary mask image. Various combinations of the spectral bands, VIs, and texture metrics and their Principal Components (PCs) were used as the input for training our candidate algorithms. In the model training phase, we used the ViT-based SegFormer architecture and the CNN-based: U-Net and DeepLab v3+ with Restnet101 backbone architecture. Furthermore, we conducted an accuracy assessment based on various evaluation metrics, which are later described in detail.

2.3. Data Retrieval and Preparation

2.3.1. Image Data

We downloaded NAIP imagery from the USGS Earth Explorer archive, managed by the USDA Farm Service Agency (FSA) through the Aerial Photography Field Office (APFO). The imagery has ground sample distances between 0.6 and 1 m and is orthorectified prior to distribution. Images are organized into uniform scenes, each potentially having various flight lines with differing geometries. Images consist of 4 spectral channels (Red, Green, Blue, and Near-infrared) (Blue (435–495 nm), Green (525–585 nm), Red (619–651 nm), and Near-Infrared (808–882 nm)) and were acquired with a Leica Geosystems ADS-100 airborne digital sensor [73]. The imagery has an 8-bit radiometric resolution. We did not perform any other processing on the NAIP imagery, as our methods were specifically designed to accommodate the diverse characteristics present within NAIP image scenes. A total of 16 NAIP image scenes were used to cover the study area. To avoid confusion in terminology, we define two key image types: "image scene" and "image patch." An "image scene" refers to the entire NAIP image downloaded from USGS Earth Explorer, typically covering a large area (3.75-minute quarter quadrangle plus a 300-meter buffer i.e., 12,596 x 12596 pixels) and containing four spectral channels (B, G, R, NIR). Image scenes are then subdivided into subarrays with a dimension of 254 x 254 pixels (152.4 m X 152.4 m) for further processing. We refer to those subarrays as "image patches" throughout the text.

2.3.2. Spectral Band Combination

We derived VIs from multispectral bands as a proxy for vegetation health. Three vegetation indices: Normalized Difference Vegetation Index (NDVI), Enhanced Vegetation Index (EVI), and Atmospherically Resistant Vegetation Index (ARVI) were created for each image patch. Since each of our candidate DL architectures takes an input image with 3 spectral channels, we used only three indices as a proxy for vegetation health. Equations 1, 2, and 3 show the formula used to calculate Vegetation indices.

$$NDVI={\displaystyle \frac{NIR-Red}{NIR+Red}}\hspace{1em}\text{[74]}$$

(1)

$$EVI=2.5\ast {\displaystyle \frac{\left(NIR–Red\right)}{\left(NIR+6\ast Red+7.5\ast Blue+1\right)}}\hspace{1em}\text{[75]}$$

(2)

$$ARVI={\displaystyle \frac{\left(NIR–\left(2\ast Red–Blue\right)\right)}{\left(NIR+\left(2\ast Red+Blue\right)\right)}}\hspace{1em}\text{[76]}$$

(3)

Similarly, we derived other 3-band combinations, such as color infrared bands i.e. (red, green, and NIR) as model inputs. Furthermore, we conducted Principal Component Analysis (PCA) on all four bands and selected the first PCA, and used corresponding red, green bands as the first two channels and PC as the third channel to create a three-band raster. With the PCA analysis, we can represent the original data in a summarized form as principal components, which maximally account for the variances from variables used [77]. We explored whether the textural properties may help in identifying vegetation health. We calculated Harlick’s Grey Level Co-occurrence Matrix (GLCM) textural metrics [78] and extracted eight standard textural features (Variance, Entropy, Contrast, Homogeneity, Angular Second Momentum, Dissimilarity, Mean, and Correlation) for each spectral band. We used the “GLCMTextures” library in R-software with a sliding window size of 3x3 at the shift angle of (1,0). Haralick texture measures could be correlated and redundant hence we employed PCA for derived metrics for each image (32 textural images for four bands) and selected the first three PCs for further analysis. Different combinations of bands used in training data (Table 1) and UTC annotations are shown in Figure 3.

2.3.3. Manual UTC Annotation

We initially identified areas heavily affected by spongy moth infestations and then randomly generated 500 sample points within these locations. NAIP scenes were then tiled into 500 image patches (254 x 254-pixel patches) centering each random point. We manually digitized all the UTCs within each image patch. We annotated a total of 400 image patches that comprise 5,133 UTCs. UTCs generally exhibit shades of grey color, however, sometimes crowns with seasonal colors (leaf senescence) may confuse human judgment. In such instances, to differentiate between unhealthy trees and those exhibiting seasonal foliage colors, we regularly consulted NDVI values and false-color band combinations. The study area consists of mature trees, mostly deciduous forests. We aimed to isolate unhealthy crowns while preserving their shape. However, dense vegetation can be tiring and confusing for human eyes, leading to inconsistencies in crown delineation. Hence, to minimize the bias, we kept the scale of 1:300 while annotating the crown. However, occluded and merged canopies posed difficulties, resulting in multiple tree crowns being annotated as a single crown as shown in Figure 4, leading to larger average crown areas.

The average crown area of annotated UTCs was around 54 m². On average each image patch contained 13 UTCs with a range of 170 UTCs and 1 UTC (Table 2). Figure 5 and Figure 6 show the distribution of UTCs per image patch and their sizes, respectively.

2.4. Deep Learning Models

2.4.1. SegFormer Model

The SegFormer [51] model consists of a transformer encoder and an MLP decoder, as shown in Figure 7.

This architecture is notable for its lightweight All-MLP decoder, which sets it apart from other transformer-based segmentation models, such as [79]. With its encoder producing multi-level feature maps that are fused in the decoder, the SegFormer can capture both high- and low-resolution information. Additionally, the model employs a "mix-FFN" operation in its encoder that replaces the original positional encoding found in other transformer architectures. Because the SegFormer is less complex than other transformer-based models, it requires less data for training and can be applied to real-time applications. Finally, the encoder can be scaled up or down, from B0 to B5, by increasing the number of layers or dimensions of encoder blocks. For this study, we used the SegFormer-B0 architecture, which has 3.75 million parameters.

2.4.2. U-Net Model

We opted for the U-Net and DeepLab v3+ CNN architectures over the pixel-wise semantic segmentation task, as they have been widely used for computer vision with computational efficiency. Initially, U-Net was devised for biomedical image segmentation tasks [31], but with time, it has been widely adopted across diverse domains including remote sensing. U-Net architecture structurally resembles a ‘U’-shape consisting of an encoding and decoding path as shown in Figure 8.

The encoder captures the semantic information from the input image. It used convolutional and pooling layers to reduce the image size while extracting increasingly complex features. The symmetrical decoder path enables the accurate location of semantic information. It uses upsampling and concatenation to increase image size gradually. U-Net has skip connections to preserve spatial information during up-sampling and down-sampling processes. While the U-Net is powerful, it can face challenges like vanishing gradients, especially in the decoder. We used ResNet101 as the encoder backbone, a deep residual network with 101 layers that has about 44.5 million parameters, to capture contextual information from the input image by mitigating the vanishing gradient problem encountered in U-Net.

2.4.3. DeepLab Model

The architectural design of DeepLab v3+ (Figure 9) encompasses an encoder-decoder path, representing an enhanced iteration of the model that integrates the encoder component from the DeepLab v3 architecture with a simplified decoder module containing around 58 million parameters. This framework leverages ResNet-101 as its foundational structure and incorporates Atrous Convolution within deep layers to expand the receptive field effectively. Subsequent to ResNet-101, an Atrous Spatial Pyramid Pooling (ASPP) module consolidates multi-scale contextual information. The decoder unit harmonizes low-level features from ResNet-101 with Upsampled deep-level multi-scale features derived from ASPP. Ultimately, it performs up-sampling of the combined feature maps to produce the definitive semantic segmentation outcomes.

2.5. DL Model Training

Candidate models were run on PyTorch, a Python-based machine learning framework that can be efficiently administered to analyze and implement DL algorithms in GPUs. We used Intel(R) Core (TM) i9-10900K CPU @ 3.70GHz processor with 128 GB RAM in a 64-bit operating system with NVIDIA® Quadro RTX 4000 with 8192MiB memory under Cuda version 12.2. To ensure the fairness of the comparative experiments, we trained three candidate models under similar conditions. For the ViT-based algorithm, we used a batch size of 12, trained for 100 epochs with early stopping, Adam optimizer was used, and the learning rate was set to 0.001 to improve the speed and effectiveness of the network. We used the weighted focal loss function, 15% weight to the background pixel whereas 85% weight to our target class to address the class imbalance problem.

2.6. Accuracy Assessment

We used standard segmentation quality metrics, such as F1- Score, Recall, and Precision, Intersection over Union (IoU), and overall accuracy used in computer vision to assess the performance of candidate algorithms.

The metric precision serves as a pivotal evaluation parameter within classification tasks, especially when confronted with imbalanced class distributions [81]. It quantifies the accuracy of a model in correctly identifying instances belonging to a specific class relative to the total instances it identifies as positive. Conceptually, precision gauges the model's capability to discern relevant instances among all instances it classifies as positive. It yields a value between 0 and 1, with a higher precision value indicating better performance. Mathematically, precision is calculated as the ratio of true positives to the sum of true positives and false positives, representing both correctly and incorrectly identified instances.

$$Precision={\displaystyle \frac{TruePositives}{TruePositives+FalsePositives}}$$

(4)

Recall, also known as sensitivity or true positive rate, is a fundamental metric in classification tasks, particularly when ensuring that all relevant instances of a class are correctly identified [81]. It quantifies the fraction of correctly labeled instances of each class relative to the total number of instances that truly belong to that class. Essentially, recall measures the model's effectiveness in capturing all relevant instances of a class among all instances that belong to that class.

$$Recall={\displaystyle \frac{TruePositives}{TruePositives+FalseNegatives}}$$

(5)

The F1-score metric is a powerful tool used to evaluate the performance of a classifier by combining precision and recall into a single value [81]. The F1-score is calculated as the harmonic mean of precision and recall. The harmonic mean is used because it punishes extreme values, making it sensitive to cases where either precision or recall is low. As a result, the F1-score tends to favor classifiers that have similar precision and recall values. Its value extends from 0 to 1, where a higher F1 score implies better overall performance in terms of both precision and recall. Mathematically,

$$F1score=2X {\displaystyle \frac{PrecisionXRecall}{Precision+Recall}}$$

(6)

Intersection over Union (IoU) is a widely used metric in semantic segmentation tasks to assess the accuracy of predicted object masks [82]. It quantifies the overlap between the predicted and ground truth masks, providing a comprehensive measure of the model's ability to accurately delineate object boundaries.

$$IoU={\displaystyle \frac{IntersectionArea}{UnionArea}}$$

(7)

The overall accuracy measures the proportion of correctly classified samples to the total number of samples [83].

$$Overallaccuracy={\displaystyle \frac{TruePositives+TrueNegatives}{TruePositives+FalsePositives+TrueNegatives+FalseNegatives}}$$

(8)

As we had a limited number of UTCs in our dataset, and our masks had more background regions than target class regions, therefore; we calculated the recall, precision, F1-score, and overall accuracy per class to gain a meaningful quantitative assessment. We then calculated the average of metric scores across both classes to provide an overall assessment of the algorithm's performance.

2.7. Manual Annotation Uncertainty Analysis

Data annotations are susceptible to bias and errors and hence, largely impact DL model performance. To address the variability in annotation and the uncertainty reflected in evaluation metrics, we engaged nine individuals to annotate 30 new image patches, each containing at least one tree crown. All annotators have sufficient academic and research background in remote sensing, hold at least a graduate-level education, and are proficient in geospatial software. They were instructed to follow the same protocol used during the generation of our initial training dataset. Furthermore, 49 UTC locations (in point form) were provided to guide them in identifying areas where annotation should be implemented. However, they were not limited to these locations; they could annotate single or multiple UTCs within the designated areas and were also free to annotate other locations outside of the guided areas based on their own judgment. We conducted a comparative analysis to understand the extent of inter-annotator variability and its potential influence on accuracy metrics. We selected 49 polygons covering our guided location points as a reference for a fair comparison. Spatial analysis was then performed to evaluate the geometric congruence of annotators, allowing us to examine differences in annotations despite the use of a standardized image, object, and protocol. First, we analyzed the annotator consensus on unhealthy pixels for the selected UTC polygons. To capture the maximum extent of UTC outline judgment across annotators, we calculated the union area of all manually outlined UTCs. To identify the minimum consensus area, we calculated the intersection of all annotated polygons.

We found that annotators produced one-to-one (Figure 10a) and one-to-many relationships in some instances (Figure 10b). Figure 10 illustrates the representative scenario of annotation we collected from annotators. For one-to-many relationship, we selected the polygon with the largest crown area for further analysis, for instance, in Figure 10b, Annotator 1 (A₁) has annotated the location as two separate UTCs and we selected the polygon with larger sizes (Figure 10 b polygon 2A₁) for calculating intersection polygon.

To evaluate the percentage of error improvement from varying annotations we calculated improvement percentages metrics (Equation 9). The metric quantifies the relative difference between the annotated and ground truth annotation, highlighting how much the annotated measurements differed from the ground truth as a percentage. Due to the lack of field-based measurements (e.g., GPS observations) of UTC extent, we used a median polygon as the baseline or ground truth against the intersection and the union to calculate the improvement percentage using the equation below:

$$\mathrm{I}\mathrm{m}\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{v}\mathrm{e}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{\%}={\displaystyle \frac{{\mathrm{M}}_x-{\mathrm{M}}_{md}\mathrm{}}{{\mathrm{M}}_{md}}}x100$$

(9)

where M_md is the median metric (baseline), and M_x is either the intersection or union metric.

3. Results

3.1. Evaluation Metrics

We evaluated candidate DL model performances under different band combinations. Table 3 depicts the test metric based on average and per class F1-score, Recall, precision, IoU, and overall accuracy. Given that the background class dominates the dataset due to the low percentage of UTC pixels, the model metrics are heavily influenced by the background class. To address this, Figure 11 provides a more meaningful comparison of test accuracy metrics across different input band combinations and candidate models, focusing on the UTC class.

3.1.1. CNN Models

The DeepLab v3+ model consistently outperforms the U-Net model across all band combinations. The U-Net exhibited the best performance for the VI band combination with an F1-score of 0.49 while the lowest performance yielded from the PCA band combination with an F1-score of 0.03. Notably, U-Net struggled with recall values below 0.50 across all combinations except for VI (0.56), indicating that it had difficulty identifying true positive UTC pixels. In contrast, DeepLab v3+ reported superior results, with F1-scores ranging from 0.38 (PCA) to 0.65 (RGB), marking an approximately more than 20% improvement over the U-Net model for similar band combinations. The RGB combination exhibited the highest F1-score (0.65), supported by a recall of 0.71 and a precision of 0.6, showcasing DeepLab v3+’s ability to balance precision and recall effectively. Despite this, the DeepLab v3+ model also showed weakness in the PCA dataset, though its F1-score of 0.38 still surpassed U-Net’s 0.04. These results highlight the consistent advantage of DeepLab v3+ over the U-Net in terms of inference accuracy, particularly with the RGB and CIR band combinations, making it the more reliable model for accurate urban tree canopy segmentation within CNN-based algorithms.

3.1.2. ViT-Based Model

Our evaluation demonstrates that the SegFormer architecture exhibits superior performance compared to conventional CNN architectures. As shown in Figure 11, the SegFormer consistently achieves F1-Scores exceeding 0.60 across most band combinations, apart from the PCA combination (F1-Score of 0.38). This suggests the suitability of the SegFormer for UTC mapping, particularly in dense forests with mixed vegetation environments. Notably, the model attains the highest F1-Score of 0.70 for the RGB band combination, indicating a well-balanced performance in terms of both accurate UTC pixel identification and minimization of false positives when using natural color images. It has a precision score of 0.62, highlighting the model’s ability to identify true positive pixels. Furthermore, the highest recall score of 0.80 suggests that it can detect most of the actual positive pixels.

In addition to F1-Score, the SegFormer consistently achieved superior IoU values across all input bands, peaking at 0.54 on the RGB band combination. It maintains IoU values of 0.48 (CIR), 0.51 (VI), 0.24 (PCA), and 0.45 (RGBPC), showing stronger results even on lower-performing bands, such as PCA and RGBPC, indicating reliable segmentation performance. Furthermore, the SegFormer’s average precision, recall, and F1-score are highest among its counterparts. This suggests that the SegFormer not only identifies UTC pixels effectively but also minimizes misclassifications across the entire image scene. The high performance of the SegFormer architecture across multiple metrics demonstrates its robustness, generalizability, and clear advantage over traditional CNN architectures in UTC extraction tasks.

The evaluation of U-Net, DeepLabv3+, and SegFormer across five band combinations (RGB, CIR, VI, PCA, RGBPC) revealed distinct trends. RGB emerged as the top-performing band for all models, with SegFormer achieving the highest IoU and F1-Score. CIR also performed well, particularly for DeepLabv3+ and SegFormer, though it still trailed RGB. SegFormer benefited the most from VI bands, while the CNN-based models, U-Net and DeepLabv3+, exhibited a decline in performance. PCA resulted in the lowest IoU across all models, especially for U-Net, which struggled significantly with this combination. However, RGBPC improved SegFormer’s performance, highlighting its ability to leverage additional features from the PCA data, whereas U-Net and DeepLabv3+ showed limited gains.

3.2. Visual Quality Assessment

We corroborated quantitative assessment with extensive visual inspections. Figure 12 shows a test image patch with a complex scene that has seasonal color (leaf senescence) on tree crowns and denser UTCs. The visualization column has input image patches, ground truth annotations, and model detection for the unseen test image patches. Visual inspection results showcase the capabilities of models to accurately detect UTCs in diverse environments. Most importantly, the differentiation between the senescent-colored tree crowns and UTCs was achieved in RGB band combination by our optimal model i.e., SegFormer (Figure 7). As seen in Figure 12, the U-Net model struggled to produce crisp boundaries around the detected objects. The outline is fuzzy and pixelated (see yellow arrows) in all the channel combinations. Among the channel combinations, the PCA and RGBPC combinations yielded the least accurate detection mask for the U-Net model, as supported by lower evaluation metrics in Figure 11. For DeepLabv3+, detections on the PCA channel showed under-segmentation, while SegFormer was over-segmented and failed to capture many UTC areas in this channel. DeepLabv3+ performed reliably on other channels but tended to have less precise UTC boundaries, suggesting it could detect the overall shapes but struggled with UTC outline accuracy. The SegFormer, however, demonstrated relatively improved boundary definition (see blue circle in the SegFormer column), offering more refined edges and accurate segmentation in complex scenes.

Figure 13 illustrates the training and validation curves for different candidate architectures across all channel combinations. DeepLabv3+ shows that it performs well on each channel, though it has varying stability for different complexities of data in the combination of its input channels. For PCA, however, a noticeable divergence between the training and validation losses later in training indicates that DeepLabv3+ tends to overfit with these combinations of channels. Similarly, the U-Net has higher initial losses, and greater fluctuations and divergence of validation loss from training loss, suggesting a struggle with generalization perhaps due to overfitting or instability when such complex or noisy data is provided. However, CIR channels are causing faster and smoother convergence in the case of U-Net, meaning that such combinations are more suitable for this model. The SegFormer has the most smooth and stable loss curves across all channels, showing its strong learning efficiency and robust generalization. Its validation loss is closer to the training loss, indicating a minimum overfitting problem. This suggests that the SegFormer is highly adaptable to various spectral combinations, and it is likely the most robust model regarding inputs of different channels.

We performed a visual comparison for one of the image patches to understand the inference capabilities and the detected mask area discrepancy of our models. In Figure 14, in each image patch, the yellow outline represents the ground truth boundary of the object, while the red outline shows the model-inferred boundary for each channel combination. DeepLabv3+ showed better alignment with the ground truth in certain channel combinations, such as RGB and RGBPC, however, it completely missed some of the smaller UTCs. The U-Net model displayed a "breadcrumb" effect in RGB and had no detection for PCA, and RGBPC channels. The SegFormer shows relatively better alignment with the ground truth in most cases, with smoother boundaries that closely follow the yellow outlines, particularly in RGB and RGBPC channels. However, in some patches, the SegFormer’s detections also slightly miss the exact boundary, either under-detection or failing to capture finer details.

Furthermore, we examined the one-to-one relationship between the masks that overlap to understand the nature of the IoU score of our detection results. Firstly, we vectorized the detections and selected only those detected polygons that intersect ground truth data (n=298). Then, we evaluated the distribution of detected and ground truth polygons by area (See Figure 15a). It was seen that detected polygons had greater area compared to ground truth polygons with mean areas of 66.83 and 50.35 square meters, respectively. There were a few instances as well where detected were smaller than ground truths (see Figure 15c). The overall mean IoU for the selected polygons was 0.52 (Figure 15b). Since these results suggested that the UTC boundary shape and size is one of the probable causes impacting mean IoU, we further investigated if the root cause is the uncertainties spawning from hand annotations. Where to draw the edge of the UTC during manual annotation is largely a judgment call. Thus, it is necessary to account for the uncertainty from annotation and examine how the model inference and evaluation metrics are impacted.

3.3. Annotation Uncertainty Analysis

Even though we had a relatively high F1-score, the IoU score seemed low. Our uncertainty analysis revealed substantial variability in annotations in both object shape and size (see Figure 16). Across 30 image patches, the number of annotated polygons ranged from 50 to 141. Interestingly, the annotator with the highest polygon count had the second lowest total annotated pixel count in the dataset. Manual annotations of a single UTC area showed considerable variation; for instance, Annotators #1 and #2 reported similar total annotated areas and pixel counts yet differed markedly in the number of individual annotations. Annotator #2 demonstrated a meticulous approach to delineating boundaries, aiming to separate individual tree crowns, and thus identified numerous small polygons as dead tree crowns, resulting in a higher number of segments than Annotator #3.

Similarly, we created intersection, median, and union polygon for each of the guided locations for all the annotators (Figure 17). The shade of the blue color shows the areas of agreement among annotators, with darker shades representing a higher level of consensus for those areas. The visual representation provides us with a clear understanding of variation across annotators. A key observation is the high level of spatial uncertainty in crown boundary delineation. While most annotators showed considerable overlap, there were notable discrepancies for complex or ambiguous features. For example, in ID 9 (Figure 17), one annotator interpreted the UTC as a large tree, resulting in a much larger annotated area compared to the others.

Furthermore, to understand the variability in IoU score, we generated an annotator-to-annotator matrix of IoU (See Figure 18). The IoU scores among the annotators varied from 0.37 to 0.70, which suggests that, despite common annotation protocols provided in the experiment, interpretations of image features can differ dramatically among annotators. The annotator-to-annotator IoU matrix underscores the challenges posed by manual annotation, especially for complex tasks like delineating tree crowns in which a given crown boundary does not provide a crisp edge to follow. Lack of strong visual cues to follow along the crown edge misleads the annotator. Such instances represent fiat object boundaries as opposed to bona fide boundaries [84] demanding the annotator to employ best approximations based on limited visual cues This observed variability indicates that, although annotation protocols can offer helpful guidance, subjective interpretation is inevitable, which can ultimately influence model performance.

Additionally, to understand the range of differences in evaluation metrics, we detected UTC segmentation on the dataset provided to the annotators using our SegFormer-RGB model. The detected masks were evaluated in relation to three reference polygon annotations: 1) intersection (maximum consensus area of all annotators), 2) union (maximum extent of all annotators), and 3) median (where most annotators reached a consensus regarding the presence of UTC pixels). The results indicate that the intersection of annotations achieved the highest recall (0.95); however, they exhibited low precision (0.23), F1-score (0.37), and IoU (0.23). In contrast, the union of annotations demonstrated high precision (0.73) alongside a mediocre recall (0.57), F1-score (0.64), and IoU (0.46). Notably, the median polygon annotations yielded the most favorable evaluation metrics, achieving high recall (0.90), F1-score (0.70), and moderate precision (0.57), with the highest IoU (0.54) among the three reference annotation types (see Table 4).

Lastly, to examine the range of improvements in model accuracy while using different annotation strategies, we calculated the relative improvements on metrics IoU, and F1-score based on Intersection and Union polygon against Median using equation (). The Results show that using an intersection polygon reduced the IoU score by 57.41 % and the F1-score by 47.14 %. In contrast, using a union polygon decreased IoU and f1-score by 14.81 % and 8.57 %, respectively.

4. Discussion

Our study compared the performance of CNN-based and ViT-based algorithms in detecting UTCs, exploring various multiband channel combinations including textural characteristics and vegetation indices. By assessing these models, we aimed to evaluate the effectiveness of publicly available aerial imagery and transformer models for detecting UTCs in large, complex environments to aid forest health monitoring. Our findings revealed significant differences in how each model interprets boundaries and spatial distributions of UTCs. Additionally, annotation uncertainty analysis emphasized the need for standardized guidelines and quality control to enhance labeling consistency.

4.1. Performance of DL Models in UTC Segmentation

Our CNN models (U-Net and DeepLab v3+) struggled to segment UTCs accurately. F1-scores ranged from 0.06 to 0.41 for U-Net and 0.23 to 0.67 for DeepLab v3+, depending on the input image channel used. This is lower than the results reported in [85], where they achieved an F1-score of 0.87 using Mask-RCNN for dead tree instance segmentation. There are a few possible reasons for this discrepancy. They had higher-resolution low-altitude drone images (20 cm) compared to our 60 cm aerial images. Our dataset contained senescent-colored foliage images, which might have introduced more noise compared to their dataset. While [85] focused on instance segmentation (identifying individual objects), we aimed for semantic segmentation (classifying each pixel as UTC or not). This difference in task complexity could also play a role. However, [86] reported similar results when using two CNN-based models (overall accuracy of 37% for AlexNet and 55% for GoogLeNet CNN models) for a region that has mixed artificial forest consisting of dead pine trees (DPTs), non-DPTs, and red broadleaf trees in RGB images. [86] claimed that confusion from color and texture features between dead pine trees and red broadleaf trees resulted in low detection accuracy. A limitation of CNN-based models is the loss of resolution when processing large images. This occurs because they downsample features to capture context, sacrificing detail around object edges. Captured from a top-down perspective, aerial images lack crucial information about tree shape and vertical structure. Furthermore, the mix of dead and dying trees, shade, dense vegetation, and senescent-colored trees adds extra semantic complexity to the segmentation task.

In Figure 12, the U-Net model exhibited a ‘salt and pepper’ effect (highlighted by yellow arrows) across all channel combinations. It might be due to channel combinations providing less distinct spectral information for differentiating target objects, leading the U-Net model to rely more heavily on minor variations and less relevant features, ultimately resulting in less coherent detections. Unlike transformer-based models such as SegFormer, which can capture global dependencies within the scene, U-Net is more prone to overfitting localized features, especially in diverse environments with significant color variation (such as leaf senescence) and dense UTCs. The DeepLabv3+ performed more reliably than U-Net across the channels but tended to have less precise UTC boundaries, suggesting it could detect the overall shapes but struggled with boundary extent accuracy. The SegFormer, however, demonstrated relatively improved boundary definition (see Figure 12, blue circle in the SegFormer column), offering more refined edges and accurate segmentation in complex scenes.

While ViT models have not been extensively studied in forest health monitoring, existing research comparing classification accuracies of CNNs and ViT on the ImageNet dataset suggests their potential for high-performance [45]. The SegFormer, the ViT-based model in our study outperformed other CNNs similar to the studies done in forestry applications [87]. He et al. 2020 [87] used HIS-BERT a transformer-based model to classify hyperspectral images and found that it outperformed all other state-of-art CNNs. The performance of the SegFormer compared to its counterpart CNNs used in our study could be attributed to the transformer’s ability to capture long-range relationships between pixels [88]. In UTC segmentation, such a relationship can be crucial for tasks, such as analyzing color variations across the entire crown, understanding the relationship between a tree and its surroundings, and contrasting UTCs to surrounding healthy canopies. While the specific reasons for this advantage require further investigation, the SegFormer’s transformer-based architecture might be particularly well-suited for analyzing high-resolution aerial imagery with complex backgrounds, potentially leading to more accurate identification of unhealthy trees. By efficiently utilizing available data and capturing relevant features from the input images, the SegFormer can generalize better to unseen data and achieve superior performance in challenging detection tasks involving complex environmental features, such as mixed tree types and varying health conditions. The attention mechanisms in Transformer models allow them to selectively focus on relevant regions of the input image while processing and preserving fine details, such as object edges and texture information, which were crucial for accurate segmentation tasks in our study. By attending to informative regions of the image, the SegFormer can effectively differentiate between different tree shapes, identify subtle cues indicative of dead or dying trees, and differentiate them with senescent-colored tree crowns in images. [88] came to a similar conclusion when used to segment forest fires with high-resolution aerial images.

The distinct patterns observed in the training and validation loss curves of DeepLabV3+, U-Net, and SegFormer illustrated the substantial influence of model architecture and input features on learning dynamics and generalization (Figure 13). For DeepLabV3+ and U-Net, fluctuating validation loss curves indicated overfitting, particularly as feature set dimensionality increases, which introduces high variance. The oscillatory nature of validation loss in the PCA channel combination indicated poor generalization capability, probably due to the limited data or noisy features. In contrast, simpler feature sets, such as CIR or RGB, resulted in more stable validation curves, suggesting a better alignment between model capacity and data variance.

The SegFormer incorporates a transformer architecture with lower inductive bias compared to CNNs [89]. It showed smoother training and validation curves across all feature sets, confirming that it captured global dependencies more effectively. It demonstrated reduced variance compared to CNN-based approaches and maintained generalization without overfitting. Overall, these findings supported the notion that transformer-based models are better for complex spatial patterns and less susceptible to overfitting than CNNs.

4.2. Performance of Band Combination in UTC Segmentation

We evaluated five different combinations of spectral data (band combinations) to train our model for identifying dead trees (UTCs) in aerial images. Surprisingly, for all candidate models none of these combinations, including those using specialized VIs, performed better than using red, green, and blue channels (RGB). This aligns with other research studies, such as [90] found that using only RGB data achieved the best results for detecting tree damage in aerial imagery. Similarly, [91] also supported our results, reporting that pre-trained models fine-tuned with RGB + NIR data underperformed compared to those using only RGB data. This performance difference is likely due to pre-trained architectures being typically trained on RGB data exclusively. Due to limited training data, we used a transfer learning strategy while keeping the backbone frozen, and only the segmentation head was trained. The feature extraction backbone was trained on standard RGB channels. Introducing spectral indices might not effectively utilize the full strength of the feature extraction backbone due to differences in data distributions. Training the encoder with large set of VIs and texture indices might be one way to capitalize on RGB channels. Also, the combination of red, green, and blue channels of natural color images offered an extensive feature space for the model. This enabled the model to learn intricate connections between pixel values and characteristics of UTCs, thereby enhancing its ability to differentiate between UTC and healthy vegetation, as well as other background elements present within the scene.

While textural information from RGB images and vegetation indices (VIs) were explored for identifying unhealthy trees, neither approach yielded optimal results. False color composites, which typically highlight unhealthy vegetation in gray, were ineffective likely due to misinterpreting noise from dead branches under healthy foliage. VIs, designed to isolate vegetation health signals mathematically, might be overly sensitive to seasonal variations in leaf color, leading to misclassifications. Textural information, particularly the smoother textures of leafless areas on dead trees, could be beneficial. However, our attempt to emphasize texture through principal component analysis (PCA) of Haralick textures proved ineffective, potentially due to information loss during dimensionality reduction. As suggested by [92], deep learning models might be better suited to directly extract information from the RGB imagery, potentially eliminating the need for pre-processing steps, such as VI calculation and PCA. However, our pre-trained model had encoders trained on the RGB channel. If we trained the encoder on VI, PCA, and texture channels, results might be expected differently.

4.3. Uncertainty and Limitations

Our study faced several challenges due to the complexity of the forest environment and data characteristics. The dense canopy covered with significant crown occlusion made it difficult to distinguish individual trees, particularly between healthy and unhealthy ones with similar spectral signatures. Acquiring data during early fall introduced additional noise, as the spectral reflectance of unhealthy trees might be similar to that of senescent-colored foliage, leading to segmentation errors. Manual annotation in such dense and visually complex scenes was time-consuming and susceptible to human bias, potentially impacting the accuracy of training data. Subtle variations in how different analysts interpreted and delineated UTC boundaries introduced inconsistencies in the training data and the model struggled to generalize to unseen data. This led to reduced accuracy on data annotated by different analysts or in different forest conditions. The chosen annotation scale (1:300), while ensuring consistent capture of sharp edges, might not capture finer details crucial for precise segmentation in occluded areas. Furthermore, the 60cm spectral resolution might be insufficient to resolve details within the dense canopy, potentially leading to spectral mixing and reduced accuracy in differentiating UTCs and might need higher resolution as emphasized by [93]. Additionally, even though models, such as SegFormer-RGB were successful in covering UTC locations, they yet exhibited uncertainty in crown boundaries. The bluntness or overestimation of boundary extent was seen in the segmented image (Figure 12). Our best-performing SegFormer-RGB model suggested that smaller ground truth UTCs tended to have more accurate and comparable detection, while larger UTCs were more prone to overestimation (see Figure 15). Although the dataset contained a fewer number of large UTCs, the over-prediction of larger UTC areas contributed to low IoU scores (Figure 15). Such performance discrepancy might be attributed to the fact that our annotation process primarily focused on delineating individual crowns, with less emphasis on accommodating noise (probably unhealthy branches inside the shadows) between annotated adjacent crowns. Consequently, this led to instances of under-segmentation during the detection phase.

Our extended investigation of uncertainties from annotations resulted in a similar conclusion as of earlier studies [94,95]. Annotators had limited consensus on judging the UTC pixels (Figure 18). However, it was clear that they were mostly biased in the edge areas of the UTC (Figure 17b and Figure 17c). This might be due to differences in domain expertise and the level of precision needed in their training data as discussed by [70]. The performance of the SegFormer-RGB model in detecting UTCs highlighted the critical role of annotation strategy in evaluating model accuracy. Our analysis compared three annotation methods—intersection, union, and median—as ground truth references, revealing significant variability in performance metrics. The intersection results showed significant variability in strict consensus areas, leading to lower precision as the model captures UTC regions only partially agreed upon by annotators. Conversely, while the union approach improved precision by considering all marked areas, it compromised recall, missing some UTCs due to less consensus. The median results emphasized that prioritizing high-consensus regions yielded the most reliable model performance, aligning detection with broader annotator agreement. A decrease in the IoU and F1-score for both intersection and union (Table 4) revealed that annotation uncertainty could impact model evaluation and underscores the importance of reaching consensus or using confidence-weighted annotations for future assessments.

Several strategies could be employed to mitigate annotation uncertainties and limitations. Despite providing a well-written protocol, variability in annotations was seen; incorporating visual examples of annotations could help improve consistency and reduce this variability [70]. Considering the limitations of our study, particularly the potential for spectral mixing due to the chosen resolution, incorporating hyperspectral information in future research could offer a more comprehensive analysis of spectral variations across different tree species. Also, incorporating a lidar-derived canopy height model could reduce uncertainty related to the boundary extent during the annotation process as well as guide the model to learn and infer the tree canopy edges. Data augmentation techniques can be used to address seasonal variations and introduce controlled variability into the training data. Similarly, adopting robust, consensus-based annotation strategies to improve the reliability of performance metrics in machine learning applications might be helpful. Future research should focus on refining the annotation process, potentially through enhanced training or semi-automated tools, to further enhance consistency and applicability in ecological contexts.

4.4. Future Outlook

While this study provides valuable insights by comparing two CNN-based models and a ViT model, it is not comprehensive. Our objective was to investigate the potential of ViT in the realm of forest monitoring applications, hence we chose simpler ViT against well-established CNNs. However, there are numerous other ViT models available today designed for segmentation tasks [96]. that can be applied in remote sensing. Future work should expand the comparative analysis to include a wider range of model architectures, as well as tasks such as instance segmentation, to rigorously validate and strengthen current findings. Furthermore, we observed that although evaluation metrics may suggest otherwise, it is not always the model's learning process that may fall short of expectations; it could also be our methodology in constructing the training data that adversely affects our metrics. Hence, beyond the annotation uncertainty analysis on evaluation metrics, future work should investigate the impact of annotation discrepancies on model training and learning behavior. By examining how models respond to noisy or ambiguous annotations, we can develop strategies to enhance their robustness and accuracy.

5. Conclusions

Our study presents a comparative analysis between CNN-based and Vision Transformer-based architectures for tree health monitoring in complicated forest scenes using publicly available aerial images. For this purpose, two established CNN architectures, U-Net and DeepLabv3+, and one Vision Transformer architecture, the SegFormer, were compared with respect to the five different band combinations. Our results revealed that, among CNN models, the DeepLabv3+ always outperformed the U-Net for UTC segmentation in all considered band combinations and natural color bands (RGB) had the best performance. However, the SegFormer outperformed both CNN-based models in different evaluation metrics like F1-score, Recall, and Precision by over 20% compared to the DeepLabv3+, and over 40% compared to U-Net.

Study findings underpin the success of Vision Transformers, particularly SegFormer, as suitable models for forest health monitoring and show how transfer learning helps boost model performance even for smaller ground truths. Notably, the RGB channel combination alone was highly effective for detecting UTCs. Also, the analysis of annotation uncertainty highlights the variability in the labeling process, emphasizing the need for improved consistency and validation. By addressing these uncertainties, the overall accuracy of the segmentation and classification models can be enhanced. Overall, the SegFormer resulted in high accuracy for the mapping of UTCs within dense and complex forested areas. However, further investigations are needed to capture unhealthy individual crowns that are partly obscured by healthy trees, which is important to study forest health. Future research should be focused on: 1) the verification of the performance of the SegFormer on different severities of crowns decline; improvement in the methods of detection of single crowns, including those hidden by canopy layers, 3) quality control measures to improve consistency and reduce variability in annotation.