Background: Validation-selected thresholds may fail to retain their apparent error-cost advantage across imaging benchmark partitions, even with satisfactory discrimination. Objective: To examine transferability of fixed, analytical cost-based, and validation-selected classification policies in PneumoniaMNIST, with threshold-selection uncertainty and sensitivity analyses. Methods: We analyzed 5,856 28 × 28 pediatric chest radiographs (4,708 training; 524 source validation; 624 testing). Eight exact train-validation overlaps were excluded from validation procedures. A 100-component logistic baseline and compact convolutional neural network (CNN) were evaluated under four hypothetical false-negative/false-positive cost ratios. Revision-stage analyses included 500 validation-threshold resamples, 1,000 paired test resamples, exact-score threshold searches, class-composition standardization, calibration diagnostics, three CNN seeds, training-duplicate removal, and a development-role-separated sensitivity experiment. Results: Logistic and temperature-scaled CNN test AUCs were 0.921 and 0.925. At equal costs, the logistic tuned-versus-analytical loss difference was −0.0192 per image, whereas the CNN loss difference was +0.0144. CNN temperature scaling increased Brier score from 0.1303 to 0.1325 and test log loss from 0.5524 to 0.5951. At a 2:1 cost ratio, the logistic validation threshold was 0.13, with a bootstrap interval of 0.05–0.63. Conclusions: Threshold-policy gains were model- and cost-dependent, and validation selection was unstable. All new comparisons are retrospective exploratory benchmark analyses; patient-level independence and clinical utility remain unverified.