Mental-health text classifiers that perform well on one dataset can degrade sharply when moved to another social platform. Researchers audited transformer models across performance, calibration, statistical significance, prediction equity, and the stability of the text features driving their decisions.
Models trained on a Kaggle corpus of 35,556 items were evaluated on 6,257 Reddit posts and 2,883 Twitter posts. Across independently evaluated models, area-under-the-curve scores fell by 30.3% to 35.4% on Reddit and 37.9% to 39.5% on Twitter compared with within-platform results.
Confidence also became less reliable. Platform-specific temperature scaling reduced the average calibration error by 88% without materially changing discrimination, indicating that confidence and classification quality failed in different ways. Important vocabulary features showed almost no overlap across platforms in most tested model-class pairs.
The labels were mapped to clinical proxies from social posts, so the work does not demonstrate diagnostic performance in healthcare. A limited fine-tuning experiment improved average accuracy, but broader validation is needed. The practical warning is clear: a mental-health model tested on one online community should not be assumed fair or accurate on another without platform-specific data and review.