Interpretability Built an Instrument to Prove a Signal Is Actually Used. Agent Memory Has Nothing Like It.

A model can contain a decodable signal and not use it to produce its answer. That sentence is the whole problem with interpretability, and it took me an embarrassingly long time to feel its weight. A ...