Decodability does not imply behavioral use, shown by amnesic probing

Elazar et al. introduce amnesic probing: apply INLP to remove a property's linear subspace from BERT-base then measure the behavioral effect on masked-LM accuracy and KL-divergence, not just whether it stays decodable [elazar-etal-2021-amnesic] Removal effects vary hugely: dependency-label removal drops LM accuracy 94%->7% and fine-POS 94%->12%, but phrase-start/end markers (probed at 85%/83%) cause essentially zero drop despite being decodable [elazar-etal-2021-amnesic] A random-direction control shows some drops are confounded with sheer dimensionality removed (dependency labels also collapse under random removal), while the phrase-marker null is genuine [elazar-etal-2021-amnesic] A selectivity control restores the gold label and fine-tunes downstream layers to check whether removal over-removed unrelated information [elazar-etal-2021-amnesic] The core message: probing accuracy and behavioral importance are dissociable in both directions, so a probed subspace being "used" needs a causal check [elazar-etal-2021-amnesic]