Robust Sentiment Analysis Through Bayesian Dropout-Enhanced RoBERTa-LSTM

Abstract

Transformer models such as RoBERTa provide strong contextualised embeddings for sentiment analysis but lack inherent mechanisms to quantify predictive uncertainty and are prone to overfitting, particularly on noisy or imbalanced text data. This paper introduces RoBERTa-LSTM-Drop, a hybrid architecture that combines RoBERTa embeddings with bidirectional LSTM layers and integrates Bayesian Dropout to capture uncertainty through Monte Carlo sampling while acting as an effective regulariser. The model is evaluated on two complementary benchmark datasets: IMDb, representing long and structured reviews, and Sentiment140, representing short and informal tweets. Comparisons are made against traditional baselines such as Logistic Regression as well as a RoBERTa-LSTM without Bayesian Dropout. Results demonstrate that RoBERTa-LSTM-B.Drop achieves accuracy competitive with strong baselines while offering calibrated uncertainty estimates that enhance model reliability. The findings highlight a stability–performance trade-off: Bayesian Dropout can yield modest accuracy improvements but introduces variance across optimisers and learning rates. Practical mitigation strategies, including dropout-rate adjustment and gradient clipping, are discussed. Beyond raw performance, uncertainty estimates enable identification of low-confidence, error-prone predictions, supporting risk-aware deployment in real-world scenarios. The analysis further clarifies when uncertainty-aware hybrids are most advantageous—such as in longer, context-rich inputs—and where their gains are limited. The work contributes both an empirical study of uncertainty-aware sentiment analysis and reproducible design choices to inform future research.