Tokamax is a library of high-performance custom accelerator kernels for JAX (on GPUs and TPUs) and PyTorch (on TPUs only), with built-in support for autotuning, shape-based heuristics, and benchmarking. For the full catalog of available operators and supported hardware backends, see Supported Ops and Hardware.
# Install dependencies if they are not already available in the runtime environment.
try:
import jax
import jax.numpy as jnp
import tokamax
import torch
import torch_tpu
except ImportError:
%pip install -q jax jaxlib tokamax torch torch_tpu
import jax
import jax.numpy as jnp
import tokamax
import torch
import torch_tpu
This Colab requires a TPU to run.
Using PyTorch on TPU? Jump straight to the PyTorch on TPU section below.
Basic Usage Example:
As an introduction to Tokamax, let's look at tokamax.ragged_dot, a drop-in replacement for jax.lax.ragged_dot.
tokamax.ragged_dot computes a Grouped Matrix Multiplication (also known as ragged matrix multiplication), which is the core computation in Mixture-of-Experts (MoE) layers. Given:
- token activations
lhsof shape(m, k)(wheremis the total number of routed tokens across all experts), - expert weights
rhsof shape(num_experts, k, n), and group_sizesof shape(num_experts,)indicating how many tokens are routed to each expert,
tokamax.ragged_dot multiplies each expert's slice of tokens by that expert's weight matrix and returns an output of shape (m, n).
import jax
import jax.numpy as jnp
import tokamax
print(f"JAX backend: {jax.default_backend()} ({jax.devices()[0].device_kind})")
def moe_loss(lhs, rhs, group_sizes):
out = tokamax.ragged_dot(
lhs,
rhs,
group_sizes=group_sizes,
preferred_element_type=jnp.bfloat16,
implementation=None, # Default: automatic selection
)
return jnp.sum(out)
moe_grad = jax.jit(jax.grad(moe_loss, argnums=(0, 1)))
JAX backend: tpu (TPU7x)
With implementation=None (the default), Tokamax automatically selects the best implementation for the current accelerator—such as Pallas:Mosaic GPU or Pallas:Triton on NVIDIA GPUs, or Pallas:Mosaic TPU on Google TPUs—and falls back to 'xla' (a pure-JAX reference implementation) when needed. You can also explicitly pass implementation='mosaic', 'triton', 'xla', or a fallback sequence like implementation=('mosaic', 'triton', 'xla').
With the forward and backward passes defined. We can now generate some tensors to feed into ragged_dot:
num_experts, m, k, n = (8, 2048, 512, 512)
key_lhs, key_rhs = jax.random.split(jax.random.key(0))
lhs = jax.random.normal(key_lhs, (m, k), dtype=jnp.bfloat16)
rhs = jax.random.normal(key_rhs, (num_experts, k, n), dtype=jnp.bfloat16)
repr_group_sizes = tokamax.generate_ragged_dot_group_sizes(
m=m,
num_groups=num_experts,
)
group_sizes = jnp.array(repr_group_sizes, dtype=jnp.int32)
out = tokamax.ragged_dot(
lhs, rhs, group_sizes=group_sizes, preferred_element_type=jnp.bfloat16
)
dlhs, drhs = moe_grad(lhs, rhs, group_sizes)
print(
"Sampled expert group sizes:",
repr_group_sizes,
f"(total={sum(repr_group_sizes)})",
)
print("Forward output shape:", out.shape, out.dtype)
print("dLHS shape:", dlhs.shape, "| dRHS shape:", drhs.shape)
Sampled expert group sizes: (231, 243, 251, 257, 271, 261, 267, 267) (total=2048) Forward output shape: (2048, 512) bfloat16 dLHS shape: (2048, 512) | dRHS shape: (8, 512, 512)
Benchmarking
Tokamax provides tokamax.benchmarking.standardize_function and tokamax.benchmarking.compile_benchmark to benchmark functions end-to-end or compare kernel implementations. Please see the benchmarking documentation for more information.
# Benchmark the full forward + backward pass using mode="forward_and_vjp"
# (You can also pass a custom jax.grad function like `moe_grad` with mode="forward".)
f_std, bench_args = tokamax.benchmarking.standardize_function(
tokamax.ragged_dot,
kwargs={
"lhs": lhs,
"rhs": rhs,
"group_sizes": group_sizes,
"preferred_element_type": jnp.bfloat16,
},
mode="forward_and_vjp",
)
run_grad = tokamax.benchmarking.compile_benchmark(f_std, bench_args)
bench_grad = run_grad(bench_args, iterations=5, method=None)
print(
f"ragged_dot (fwd+bwd) | median: {bench_grad.median_evaluation_time_ms:.3f}"
f" ms | compile: {bench_grad.compile_time_ms:.1f} ms | peak mem:"
f" {bench_grad.peak_memory_mb} MB"
)
# Compare TPU implementation vs. XLA fallback
for impl in ("mosaic", "xla"):
impl_std, impl_args = tokamax.benchmarking.standardize_function(
tokamax.ragged_dot,
kwargs={
"lhs": lhs,
"rhs": rhs,
"group_sizes": group_sizes,
"preferred_element_type": jnp.bfloat16,
"implementation": impl,
},
mode="forward",
)
run_impl = tokamax.benchmarking.compile_benchmark(impl_std, impl_args)
bench_impl = run_impl(impl_args, iterations=5, method=None)
print(
f"implementation={impl!r:<6} | median:"
f" {bench_impl.median_evaluation_time_ms:.3f} ms"
)
ragged_dot (fwd+bwd) | median: 0.084 ms | compile: 558.0 ms | peak mem: 25.189408 MB implementation='mosaic' | median: 0.038 ms implementation='xla' | median: 0.014 ms
Kernel Configs: Heuristics & the Autotuning Cache
Each Tokamax kernel is parameterized by a config: a backend-specific set of tuning knobs (such as tile sizes) that controls how the kernel's work is mapped onto the accelerator. The best config depends on the input shapes and dtypes, the hardware, and (for ragged_dot) the distribution of group sizes. Tokamax resolves each kernel's config in the following order:
- Autotuning Cache: Checks any active
AutotuningResultcontext (see Autotuning below) first, followed by Tokamax's built-in persistent autotuning cache of pre-tuned configs for popular model shapes on the current accelerator. - Cache-Miss Fallback (
tokamax.config.autotuning_cache_miss_fallback): Defaults to'heuristics', which computes a hardware-aware heuristic config from the input shapes, dtypes, representativeRaggedDotGroupSizes, and device memory limits (e.g., TPU VMEM or GPU shared memory). It can also be set to'autotune'(autotune on a cache miss) or'error'.
Configs are chosen at trace time. Tokamax resolves configs while JAX traces your function (for example, on the first call to a jax.jit-compiled function) and bakes them into the compiled program. Tokamax's config contexts, such as tokamax.config.ignore_autotuning_cache(...) and with autotune_result:, are part of JAX's jit cache key: calling an already-compiled function (like moe_grad above) inside one of them retraces it with the new configs, while calls outside the context keep using the default configs. Keep the context active around every call that should use it.
Manually Setting Tokamax Op Configs
High-level functions like tokamax.ragged_dot automatically select the backend implementation and resolve kernel configurations via the autotuning cache or heuristics, as described above.
When you need low-level control over kernel hyperparameters (such as tile sizes), you can instantiate a backend-specific Op implementation class directly and pass a manual Config. Note that because Config parameters are hardware-specific, importing a backend implementation class (such as RaggedDot from tokamax.ops.tpu below for TPU) targets that specific accelerator, and overriding tile sizes can significantly affect performance:
import tokamax
from tokamax.ops.tpu import RaggedDot, RaggedDotConfig
num_experts, m, k, n = 8, 2048, 512, 512
key_lhs, key_rhs = jax.random.split(jax.random.key(0))
lhs = jax.random.normal(key_lhs, (m, k), dtype=jnp.bfloat16)
rhs = jax.random.normal(key_rhs, (num_experts, k, n), dtype=jnp.bfloat16)
group_sizes = jnp.full((num_experts,), m // num_experts, dtype=jnp.int32)
# Option 1: High-Level API
# Automatically selects the best supported backend and pulls configs from the
# autotuning cache or heuristics.
out = tokamax.ragged_dot(
lhs,
rhs,
group_sizes=group_sizes,
preferred_element_type=jnp.bfloat16,
implementation=None,
)
# Option 2: Op Implementation Class
# Instantiate the backend-specific Op class with a manual Config to override
# kernel tile sizes.
ragged_dot_config = RaggedDotConfig(tile_m=256, tile_k=256, tile_n=256)
ragged_dot_op = RaggedDot(config=ragged_dot_config)
out = ragged_dot_op(
lhs,
rhs,
group_sizes=group_sizes,
preferred_element_type=jnp.bfloat16,
)
Benchmarking a Suboptimal Manual Config vs. the Heuristic Config
Choosing inappropriate tile sizes in a manual Config can substantially degrade performance. For example, setting very small tile_m, tile_k, and tile_n tiles (128 x 128 x 128) underutilizes TPU VMEM and the Matrix Multiply Units (MXU) while issuing many small DMA transfers. By contrast, the built-in heuristic config automatically sizes tiles to fit VMEM and align with MXU dimensions. The example below reuses RaggedDot, RaggedDotConfig, and the lhs, rhs, and group_sizes inputs from the manual config example above:
# 1. Suboptimal manual config: tiny (128 x 128 x 128) tiles
slow_config = RaggedDotConfig(tile_m=128, tile_k=128, tile_n=128)
slow_op = RaggedDot(config=slow_config)
# 2. Hardware-aware heuristic config: sizes tiles to fit TPU VMEM & MXU
heuristic_config = (
RaggedDot()
.bind(lhs, rhs, group_sizes=group_sizes)
.heuristics_config
)
heuristic_op = RaggedDot(config=heuristic_config)
for cfg_label, rd_op, cfg in [
("Manual (slow config)", slow_op, slow_config),
("Heuristic (fast config)", heuristic_op, heuristic_config),
]:
rd_std, rd_args = tokamax.benchmarking.standardize_function(
rd_op,
kwargs={
"lhs": lhs,
"rhs": rhs,
"group_sizes": group_sizes,
"preferred_element_type": jnp.bfloat16,
},
mode="forward",
)
run_rd = tokamax.benchmarking.compile_benchmark(rd_std, rd_args)
bench_rd = run_rd(rd_args, iterations=5, method=None)
print(
f"{cfg_label:<24} | median: {bench_rd.median_evaluation_time_ms:.3f} ms |"
f" config: tile_m={cfg.tile_m}, tile_k={cfg.tile_k}, tile_n={cfg.tile_n}"
)
Manual (slow config) | median: 0.106 ms | config: tile_m=128, tile_k=128, tile_n=128 Heuristic (fast config) | median: 0.038 ms | config: tile_m=1024, tile_k=512, tile_n=512
Autotuning
Heuristic configs are a good default, but to get peak performance for your workload's exact shapes and group sizes, autotune the Tokamax kernels in a function using tokamax.autotune(fn, *args), which benchmarks candidate configs for each kernel and records the fastest.
- The returned
AutotuningResultholds the fastest configuration for the exact input shapes and dtypes passed totokamax.autotune. - Use it as a
with autotune_result:context manager to apply those tuned configurations to matching kernel calls traced inside the block. Like the other config contexts, it is part of JAX's jit cache key, so keep it active around every call that should use the tuned configs (for example, your whole training loop). - Serialize it to JSON via
.dumps()and reload it later (for example, in your training job) viatokamax.AutotuningResult.loads(). This is useful when autotuning each op may take a long time so it is wiser to run it once and cache the results.
# Note: Autotuning is specific to the exact input shapes and dtypes. In your own
# workloads, pass your full production shapes (e.g., lhs, rhs, group_sizes).
# Here we use a smaller (m, k, n) = (128, 128, 128) shape so this demo cell
# finishes quickly:
lhs_tune = lhs[:128, :128]
rhs_tune = rhs[:, :128, :128]
repr_gs_tune = tokamax.generate_ragged_dot_group_sizes(
m=128, num_groups=num_experts
)
gs_tune = tokamax.RaggedDotGroupSizes(
jnp.array(repr_gs_tune, dtype=jnp.int32), repr_gs_tune
)
autotune_result: tokamax.AutotuningResult = tokamax.autotune(
moe_grad, lhs_tune, rhs_tune, gs_tune
)
# Calls traced inside the context use the tuned configs:
with autotune_result:
dlhs_autotuned, drhs_autotuned = moe_grad(lhs_tune, rhs_tune, gs_tune)
# Serialize and reload the autotuning result. This serialized result can be
# written out to disk.
autotune_result_json: str = autotune_result.dumps()
autotune_result = tokamax.AutotuningResult.loads(autotune_result_json)
print(f"Autotuned {len(autotune_result.data)} kernel calls.")
Autotuning: 100%|██████████| 2/2 [00:01<00:00, 1.01 op calls/s, Total microbenchmarks=6] Autotuned 2 kernel calls.
Tokamax PyTorch on TPU
Tokamax's Pallas Mosaic TPU kernels are bridged into PyTorch via torch_tpu and exposed under tokamax.experimental.torch_tpu.ops. Importing torch_tpu registers the "tpu" device in PyTorch, allowing you to allocate tensors with device="tpu" and invoke Tokamax kernels directly on torch.Tensor inputs without host copies or writing JAX code.
For the full list of operators with TorchTPU API support, see Supported Ops and Hardware. Note that online autotuning (tokamax.autotune) and benchmarking are currently only supported in JAX, though PyTorch operators can load pre-autotuned configurations from Tokamax's autotuning cache.
JAX vs. PyTorch at a Glance
If you already know the JAX API above, here are the key differences in the PyTorch API:
JAX (tokamax) |
PyTorch (tokamax.experimental.torch_tpu.ops) |
|
|---|---|---|
| Hardware | GPUs and TPUs | TPUs only (experimental) |
| Inputs | jax.Array |
torch.Tensor on device="tpu" |
| Gradients / compilation | jax.grad / jax.jit |
torch.autograd / torch.compile |
| Default kernel config | Autotuning cache, falling back to heuristics | Heuristics (opt into the cache with get_configs(..., from_autotuning_cache=True)) |
| Manual kernel config | Instantiate a backend Op class with a Config |
Pass configs=(fwd_config, bwd_config) |
| Autotuning and benchmarking | ✅ tokamax.autotune, benchmarking |
❌ Not yet supported |
Eager Execution and Autograd
Tokamax PyTorch operators can be called just like standard torch.nn.functional operations on device="tpu" tensors and integrate seamlessly with torch.autograd. Here is how to run the forward and backward passes of tokamax_torch.ragged_dot in eager mode:
import tokamax.experimental.torch_tpu.ops as tokamax_torch
import torch
import torch_tpu # Registers the "tpu" device in PyTorch
num_experts, m, k, n = 4, 512, 256, 512
lhs = torch.randn(m, k, device="tpu", dtype=torch.bfloat16, requires_grad=True)
rhs = torch.randn(
num_experts, k, n, device="tpu", dtype=torch.bfloat16, requires_grad=True
)
group_sizes = torch.tensor(
[128, 128, 128, 128], device="tpu", dtype=torch.int32
)
out = tokamax_torch.ragged_dot(lhs, rhs, group_sizes=group_sizes)
loss = out.sum()
loss.backward()
grad_lhs = lhs.grad
grad_rhs = rhs.grad
Compiling with torch.compile
Because every Tokamax operator is registered with torch.library and register_autograd, you can include it directly inside @torch.compile regions alongside standard PyTorch operations:
@torch.compile(fullgraph=True, dynamic=False)
def train_step(
lhs: torch.Tensor, rhs: torch.Tensor, group_sizes: torch.Tensor
) -> torch.Tensor:
lhs = lhs + 1.0
out = tokamax_torch.ragged_dot(lhs, rhs, group_sizes=group_sizes)
return out.sum()
lhs.grad = None
rhs.grad = None
loss = train_step(lhs, rhs, group_sizes)
loss.backward()
Kernel Configurations and Autotuning Cache
To keep torch.compile fast and predictable, Tokamax PyTorch operators use shape-based heuristics by default rather than reading the on-disk autotuning cache implicitly. Passing configs to an operator is optional—omitting configs automatically selects heuristic configurations for your input shapes.
If you want to inspect configurations ahead of time or load pre-autotuned configs from cache, call tokamax_torch.tokamax_torch_utils.get_configs(op, *args, from_autotuning_cache=...), which returns (forward_config, backward_config) (or (forward_config, None) for forward-only ops):
- Heuristics (
from_autotuning_cache=False, default): Computes configurations from the input shapes and dtypes. - Autotuning Cache (
from_autotuning_cache=True): Looks up offline-autotuned configurations for the target TPU and input shapes, falling back to heuristics on a cache miss.
Here is an example with tokamax_torch.ragged_dot:
lhs = torch.randn(
512, 256, device="tpu", dtype=torch.bfloat16, requires_grad=True
)
rhs = torch.randn(
4, 256, 512, device="tpu", dtype=torch.bfloat16, requires_grad=True
)
group_sizes = torch.tensor(
[128, 128, 128, 128], device="tpu", dtype=torch.int32
)
# Get forward and backward configs from the heuristics (default).
fwd_config, bwd_config = tokamax_torch.tokamax_torch_utils.get_configs(
tokamax_torch.ragged_dot,
lhs,
rhs,
group_sizes=group_sizes,
from_autotuning_cache=False,
)
out = tokamax_torch.ragged_dot(
lhs,
rhs,
group_sizes=group_sizes,
configs=(fwd_config, bwd_config),
)
When group_sizes is passed to get_configs() as a torch.Tensor, get_configs() only inspects tensor shape and dtype metadata (assuming a uniform split of m tokens across experts), so you can resolve configurations at model initialization time using device="meta" tensors without allocating TPU memory. (If you instead pass a Python tuple or list of representative token counts, get_configs() uses that exact distribution for config selection, as shown in the next section.)
lhs_meta = torch.empty((262144, 7168), dtype=torch.bfloat16, device="meta")
rhs_meta = torch.empty((8, 7168, 2048), dtype=torch.bfloat16, device="meta")
group_sizes_meta = torch.empty((8,), dtype=torch.int32, device="meta")
fwd_config, bwd_config = tokamax_torch.tokamax_torch_utils.get_configs(
tokamax_torch.ragged_dot,
lhs_meta,
rhs_meta,
group_sizes=group_sizes_meta,
from_autotuning_cache=True,
)
Putting it all together, here is the PyTorch equivalent of the ragged_dot example shown at the top of the Colab.
import tokamax
import tokamax.experimental.torch_tpu.ops as tokamax_torch
import torch
import torch_tpu # Registers the "tpu" device in PyTorch
tpu_device = torch.device("tpu")
num_experts, m, k, n = (8, 2048, 512, 512)
torch.manual_seed(0)
lhs_torch = torch.randn(
m, k, device=tpu_device, dtype=torch.bfloat16, requires_grad=True
)
rhs_torch = torch.randn(
num_experts,
k,
n,
device=tpu_device,
dtype=torch.bfloat16,
requires_grad=True,
)
# Generate a realistic skewed (Zipf-like) MoE expert group sizes distribution:
raw_weights = [i**-1.2 for i in range(1, num_experts + 1)]
expert_probs = [w / sum(raw_weights) for w in raw_weights]
repr_group_sizes = tokamax.generate_ragged_dot_group_sizes(
m=m, num_groups=num_experts, p=expert_probs, seed=42
)
group_sizes_torch = torch.tensor(
repr_group_sizes, device=tpu_device, dtype=torch.int32
)
# Optional: retrieve forward & backward configs using the representative
# group sizes tuple (pass from_autotuning_cache=True to check the cache first):
fwd_config, bwd_config = tokamax_torch.tokamax_torch_utils.get_configs(
tokamax_torch.ragged_dot,
lhs_torch,
rhs_torch,
group_sizes=repr_group_sizes,
from_autotuning_cache=False,
)
# Run forward and backward passes with PyTorch autograd inside @torch.compile:
@torch.compile(fullgraph=True, dynamic=False)
def torch_moe_step(lhs_in, rhs_in, gs_in):
return tokamax_torch.ragged_dot(
lhs_in,
rhs_in,
group_sizes=gs_in,
preferred_element_type=torch.bfloat16, # Output/accumulation dtype
configs=(
fwd_config,
bwd_config,
), # Optional: omit to use default heuristics
)
out_torch = torch_moe_step(lhs_torch, rhs_torch, group_sizes_torch)
loss_torch = out_torch.sum()
loss_torch.backward()
print(
"Sampled expert group sizes:",
repr_group_sizes,
f"(total={sum(repr_group_sizes)})",
)
print("Forward config:", fwd_config)
print(
"PyTorch output shape:",
tuple(out_torch.shape),
out_torch.dtype,
out_torch.device,
)
print(
"dLHS grad shape:",
tuple(lhs_torch.grad.shape),
"| dRHS grad shape:",
tuple(rhs_torch.grad.shape),
)
Sampled expert group sizes: (869, 389, 209, 183, 131, 98, 93, 76) (total=2048) Forward config: Config(tile_m=None, tile_k=None, tile_n=None, bucket_base=None, lhs_buffer_count=None) PyTorch output shape: (2048, 512) torch.bfloat16 tpu:0 dLHS grad shape: (2048, 512) | dRHS grad shape: (8, 512, 512)