Debugging data non-determinism

This set of documentation describes debugging techniques when your TPU programs do not work as expected. This can include program non-determinism in outputs or observable intermediate results, unexpected NaNs/Inf, wrong outputs, and more.

The localization and debugging workflow involves amending XLA code and rerunning the reproducer with new recompiled binary, similar to on-device debugging workflow.

Common symptoms & localization approach

For large-scale TPU program debugging the localization is usually the most complicated part, involving different techniques depending on the category of symptoms observed, most common ones are: