Non-determinism in programs can occur due to a bad chip or a bug in XLA (both HLOs and kernels). The XLA SDC Checker (XSC) can detect non-determinism in all of these cases. It reports them as SDC events that need to be analyzed to localize the cause of non-determinism.
Basic configuration
Enable XSC Checker with the following flag. By default, it instruments the most non-trivial HLOs to run the HLO twice and check if the output checksum is the same for both runs.
--xla_tpu_enable_sdc_checker=true
You can see the list of instrumented HLOs in the compilation logs, including the reason of non-instrumenting, for example:

SDC Events analysis: Bad chips or XLA bugs
All detected non-determinism is reported as SDC events, and you need to distinguish between the events indicating bad chips from those indicating XLA/kernel bugs. The best way to do so is to collect statistics over many events and reason about event locality:
- If all collected events are identified at the same chip (rarely few chips) and happen in various HLOs/kernels, it most probably indicates hardware issues.
- If the events are distributed evenly across the chips in the slice, but
mostly happen in the same/similar HLOs (e.g. the similar
AllReduceHLOs, or similar kernels), then it may indicate some bug in those HLOs.
More configurations
The following configurations can be helpful for detecting more non-determinism events and localizing them:
- Silent reporting: It is
recommended to use "silent reporting mode", where XSC does not crash the
program but allows it to continue execution after reporting SDC event
information to the log. Set the flag:
--xla_tpu_sdc_check_halt_on_detection=false - Increase number of HLO repeats: It may be
beneficial in some cases to increase the number of HLO reruns by setting
N>1:
--xla_tpu_sdc_check_repeat_count=N - Filtering instrumented HLOs: Make
debugging more focused on particular HLO kinds or particular set of HLOs.
Set the flag:
--xla_tpu_sdc_checker_extra_filter=...
If the basic configuration does not find any events: Compile your program without deduplicated HLOs.
Depending on your program, also see:
SparseCore HLO rerun
Some of SparseCore programs, namely offloaded collectives, also support HLO rerun directly on SparseCore. TensorCore instrumentation does NOT support rerun of *-start or *-done of the offloaded collective, so rerunning the operation on SparseCore is the only way to reproduce non-determinism.
SparseCore instrumentation is enabled with the following flag.
--xla_tpu_enable_sc_sdc_checker=true
The number of reruns may be changed with the following flag. Not specifying this flag results in XSC performing the same number of operations on SparseCore, as it does on TensorCore.
--xla_tpu_enable_sc_sdc_checker_loop_iteration_count=N
Instrumenting custom kernels
Sometimes kernels implementation does not comply with requirements of determinism XSC assumes. For example, in case of ragged kernels, some part of the output tensor is intentionally left uninitialized.
To address this problem XSC forces initialization of all kernel outputs with 0 value, which causes extra overhead and may hide some of the issues. Even with zeroing output buffers, custom kernels' may still produce non deterministic results for any reason the author of the kernel may have. Sometimes it is beneficial to disable all kernel implementation:
--xla_tpu_sdc_checker_enable_tpu_custom_calls=false
If there are just a few kernels which should be disabled, more flexible HLO name filtering can be used.
Double-checksum
XSC has an option to compute each of the checksum twice in a row and see if it has changed across the computations. This checksum can be enabled with:
--xla_tpu_sdc_checker_double_check_checksums='true'
An event detected here would mean that the buffers are corrupted at the moment of checksumming. Such corruption may be a result of the HLO being executed or any other HLO executed concurrently to it.
Usually this feature is combined with enabling checksumming of the HLO inputs:
--xla_tpu_sdc_check_inputs=true
And, disabling HLO rerun to allow the XSC to instrument more HLOs (i.e. asynchronous operations which cannot be instrumented when N>0):
--xla_tpu_sdc_check_repeat_count=0
Performance impact of the instrumentation
Rerunning HLOs and checksumming inputs and outputs comes at cost. Depending on how many HLOs are instrumented and how big checksummed buffers are the performance impact of the instrumentation may be significant.
With checksumming only outputs and only 1 HLO rerun, on an average model you can expect performance overhead of 1.8-2.5x. For debugging, it is usually not a problem unless extra execution time impacts reproducibility of the issue being debugged.
Flags reference
--xla_tpu_enable_sdc_checker=true: enables instrumentation on TensorCore--xla_tpu_sdc_check_repeat_count: the number of HLO reruns, default: 1 (reruns HLO once after the primary HLO execution)--xla_tpu_sdc_check_halt_on_detection: if false, enables silent reporting mode, default: true--xla_tpu_enable_sc_sdc_checker='true': if xla_tpu_enable_sdc_checker is set also enables checks on SparseCore--xla_tpu_enable_sc_sdc_checker_loop_iteration_count: specifies number of reruns on SparseCore, default: value xla_tpu_sdc_check_repeat_count--xla_tpu_sdc_check_inputs: if the inputs of the HLO need to be checksummed, default: false--xla_tpu_sdc_checker_enable_tpu_custom_calls: if the kernels need to be instrumented, default: true--xla_tpu_sdc_checker_checksum_algo: specifies checksum algorithm to be used, values: {'default', 'xor', 'siphash13'}, default: 'xor'--xla_tpu_sdc_checker_double_check_checksums: enables double-checksumming, default: false--xla_tpu_sdc_checker_extra_filter=...--xla_tpu_disable_aliasing_extra_filter=...--xla_jf_hlo_deduplication_all_unique=true--xla_jf_hlo_deduplication_all_unique_filter=...