This section includes a collection of guides and reference docs for debugging TPU programs. Both JAX and TPU-SDK provide an extensive set of debugging tools and techniques, and choosing the right one is a technical decision itself.
Debugging can be divided into several levels, where each group offers diagnostics for a specific topic or area.
JAX
When starting debugging your JAX model, JAX offers a high-level and intuitive API and flags for logging and troubleshooting your model. If you are just starting, consider it as your first choice for any investigation, before jumping directly to a more complex tooling:
HLO debugging
For TPU, the JAX program is lowered to HLO, which can be extensively inspected for errors. HLO dumps provide an insight into generated modules:
Domain-specific debugging
For a range of domain-specific errors, check out the following documentation on some common cases:
- 01 - Megascale Error Handling & Hangs
- 02 - Megascale Stragglers
- 03 - Megascale Performance
- 04 - OOM + HBM OOM (Error code 1000)
- 05 - TPU Program On-device Debugging
- 06 - Reporting JAX/XLA Compile-time Crashes on TPU
- 07 - Pallas/Mosaic Kernel Debugging (coming soon)
- 08 - Data Non-determinism (coming soon)
- 09 - Numerics-related Errors (coming soon)
- 10 - SparseCore Debugging Tools
- 11 - HLO & LLO Dumps