Anatomically Faithful but Temporally Diffuse: Auditing Attribution for Left-Ventricular Ejection-Fraction Estimation from Echocardiography
Deep video models estimate left-ventricular ejection fraction (EF) from echocardiography with near-expert accuracy, and post-hoc attribution is increasingly used to certify that such models look at the right place. Because EF is defined by the end-systolic (ES) and end-diastolic (ED) frames, a faithful explanation must localize not only the left ventricle in space but also the decisive frames in time. We audit attribution faithfulness along three axes -- spatial grounding, perturbation, and temporal reliance -- for two architecturally distinct regressors fine-tuned on EchoNet-Dynamic: a self-supervised VideoMAE transformer audited with Chefer relevance propagation, and a Kinetics-pretrained R(2+1)D convolutional network audited with Grad-CAM. All statistics are reported with bootstrap 95% confidence intervals over the full 1,276-study test split, against explicit chance baselines and attention-rollout and random-attribution controls. We find a pronounced asymmetry rather than a temporal null. Both models are strongly anatomically faithful (3.04x and 3.76x chance) but only weakly localized in time: tubelet occlusion places causal reliance on ES/ED at 1.05x chance for VideoMAE (95% CI [1.037, 1.068]) and 1.15x for R(2+1)D ([1.134, 1.166]) -- a threefold weaker concentration than in space -- and the attribution faithfully mirrors this weakness. Separating convergence from architecture shows that temporal reliance is architecture-determined and convergence-invariant: R(2+1)D attains 1.14x after a single training epoch and 1.15x at convergence, while both VideoMAE variants stay at 1.05x regardless of pretraining or accuracy. Spatial faithfulness therefore does not imply temporal faithfulness, and the shortfall is not remedied by further training.