RenalTCN: A Hybrid Transformer-Convolutional Architecture with Composite Loss Functions for Robust Kidney Tumor Segmentation

Abstract

Segmenting tumors from computed tomography (CT) scans of the kidneys can help to guide surgical procedures, monitor treatment efficacy, and to predict outcomes for patients with renal cell carcinoma (RCC). However, the extreme foreground–background class imbalance, wide morphological heterogeneity of renal masses, and the visual overlap between benign cysts and malignant tumors prevents the accurate diagnosis of kidney tumors. In this paper, RenalTCN, a hybrid network formed by combining a convolutional encoder-decoder with a transformer bottleneck to learn local fine details and anatomical features. Pure conventional networks are efficient for capturing local details but fail at capturing global details. Conversely, transformer-based networks capture long-range dependencies at the expense of fine spatial precision and computational efficiency. RenalTCN combines both convolutional network and transformer models to overcome the difficulties faced by both the models. The present model is supervised with auxiliary outputs, a well-designed weight decay for focal, Dice and cross-entropy losses to overcome the dramatic class imbalance. Additionally, this approach can be helpful in practical optimisations, gradient accumulation, mixed precision, and cosine annealing using differentiated learning rate to stabilise and accelerate convergence. On the validation set, RenalTCN achieved Dice scores of 0.965 ± 0.02 for the kidney and 0.820 ± 0.08 for the tumor. It has been shown to be consistent across the size of the tumor, with good sensitivity of 90% for small tumors (less than or equal to 4 cm), 100% for medium-size (4–7 cm) and 96% for large tumors (>7 cm). The ablation studies demonstrated that each design decision is crucial and removing the Transformer bottleneck alone resulted in the most significant drop in Dice score (4.0%), indicating the most significant drop in Tumor Segmentation accuracy. In general, RenalTCN strikes a sensible compromise between model complexity and computational cost and its training framework directly addresses the two primary issues of class imbalance and optimisation instability, making it feasible for real-world clinical scenarios that have limited hardware resources.