Probing accuracy collapses under a naturalistic counterfactual test

Amini et al. compare correlational probing for grammatical gender/number against evaluation on naturalistic counterfactual test pairs [amini-etal-2023-naturalistic] Probes reaching above 90% on the standard split drop to roughly 60% (near chance for a binary property) on dependency-tree-verified counterfactual sentences [amini-etal-2023-naturalistic] Standard probing accuracy on an uncontrolled test distribution can substantially overstate how reliably a property is linearly encoded [amini-etal-2023-naturalistic] This independently confirms the decodability-vs-behavioral-importance caution from amnesic probing and completeness/selectivity frameworks [amini-etal-2023-naturalistic]