Operator Fusion
tokamax.ops.op_fusion lets you fuse surrounding JAX operations (prologues and
epilogues) directly into a fusible Tokamax Pallas TPU kernel via
JAX Pallas Fuser.
Instead of materializing intermediate activations to HBM between a normalization
or activation and a custom kernel, the fused operations run in VMEM on each tile
inside the kernel loop.
Using @op_fusion.fuse
Basic Usage (Prologue and Epilogue)
Import op_fusion from tokamax.ops and wrap the function containing a fusible
Tokamax operator with @op_fusion.fuse (typically inside @jax.jit):
import jax
import jax.numpy as jnp
import tokamax
from tokamax.ops import op_fusion
def rms_norm(x: jax.Array, weight: jax.Array, eps: float = 1e-6) -> jax.Array:
ms = jnp.mean(jnp.square(x.astype(jnp.float32)), axis=-1, keepdims=True)
return (x.astype(jnp.float32) * jax.lax.rsqrt(ms + eps)).astype(
x.dtype
) * weight
@jax.jit
@op_fusion.fuse
def fused_ragged_dot(x, rms_weight, rhs, out_bias, group_sizes):
# Prologue: fused into the kernel's LHS tile load.
x_norm = rms_norm(x, rms_weight)
out = tokamax.ragged_dot(
x_norm,
rhs,
group_sizes=group_sizes,
implementation="mosaic_tpu_v2",
)
# Epilogue: fused into the kernel's output store.
return jax.nn.silu(out + out_bias)
You can call either the high-level functional API with an explicit fusible
implementation (tokamax.ragged_dot(..., implementation="mosaic_tpu_v2")) or a
specific tokamax.Op instance directly (such as
PallasMosaicTpuV2RaggedDot).
If an operator inside @op_fusion.fuse does not support fusion, Tokamax raises
NotImplementedError.
Decorator Options
op_fusion.fuse forwards its keyword arguments to
jax.experimental.pallas.fuser.fuse:
@op_fusion.fuse(
resolve_fusion_dtypes=True,
debug=False,
strict_mode=True,
static_argnums=None,
static_argnames=None,
)
static_argnums/static_argnames: Mark non-array Python arguments (for example, scalar config values or boolean flags) as static when calling a fused function.debug=True: Prints the traced fusion jaxprs to help diagnose why an op did not fuse or failed block-spec propagation.strict_mode: Controls Pallas Fuser's strict block-spec equivalence checks.
Quantized Inputs and Built-In Kernel Activations
Fused prologues compose with on-the-fly activation quantization (AsQArray),
pre-quantized weights (qwix.QArray), and built-in fused gate/up activations on
PallasMosaicTpuV2RaggedDot:
import jax
import jax.numpy as jnp
import qwix
from tokamax._src import quantization
from tokamax._src.ops.ragged_dot import pallas_mosaic_tpu_v2
from tokamax.ops import op_fusion
gmm_op = pallas_mosaic_tpu_v2.PallasMosaicTpuV2RaggedDot()
@op_fusion.fuse
def fused_rms_norm_gate_up(x, rms_weight, w_q, w_scale, group_sizes):
x_norm = rms_norm(x, rms_weight)
return gmm_op(
quantization.AsQArray(x_norm, jnp.float8_e4m3fn),
qwix.QArray(w_q, w_scale),
group_sizes=group_sizes,
zero_initialize=False,
fuse_gateup_activation="silu",
preferred_element_type=jnp.bfloat16,
)
One Fusible Op per @op_fusion.fuse Block
JAX Pallas Fuser supports a single fusible kernel per @op_fusion.fuse scope.
When chaining multiple kernels (for example, gate_up followed by down in an
MoE block), wrap each fusible stage in its own @op_fusion.fuse function inside
an outer @jax.jit:
@jax.jit
def moe_layer(x, rms_weight, w1_q, w1_scale, w2_q, w2_scale, group_sizes):
mid = fused_rms_norm_gate_up(x, rms_weight, w1_q, w1_scale, group_sizes)
return gmm_op(
quantization.AsQArray(mid, jnp.float8_e4m3fn),
qwix.QArray(w2_q, w2_scale),
group_sizes=group_sizes,
zero_initialize=False,
preferred_element_type=jnp.bfloat16,
)
Automatic Differentiation with Explicit @jax.custom_vjp
@op_fusion.fuse fuses the forward computation block and does not automatically
synthesize fused backward passes through jax.grad. When differentiating a
fused block, wrap the forward and backward calls in an explicit
@jax.custom_vjp so you can fuse prologues/epilogues into the forward and
backward (drhs / dlhs) kernels independently:
import jax
from tokamax._src.ops.ragged_dot import pallas_mosaic_tpu_v2
from tokamax.ops import op_fusion
op = pallas_mosaic_tpu_v2.PallasMosaicTpuV2RaggedDot()
drhs_op = pallas_mosaic_tpu_v2.PallasMosaicTpuV2RaggedDot(
num_actual_groups=num_groups
)
@op_fusion.fuse
def fused_fwd_kernel(x, w, r, group_sizes):
return op(rms_norm(x, w), r, group_sizes=group_sizes, zero_initialize=True)
@op_fusion.fuse
def fused_drhs_kernel(x, w, dy, group_sizes):
return drhs_op(
rms_norm(x, w),
dy,
group_sizes=group_sizes,
ragged_dot_dimension_numbers=pallas_mosaic_tpu_v2.DRHS_RAGGED_DOT_DIM_NUMS,
preferred_element_type=dy.dtype,
)
@jax.custom_vjp
def fused_gmm(x, w, r, group_sizes):
return fused_fwd_kernel(x, w, r, group_sizes)
def fused_gmm_fwd(x, w, r, group_sizes):
return fused_fwd_kernel(x, w, r, group_sizes), (x, w, r, group_sizes)
def fused_gmm_bwd(residuals, dy):
x, w, r, group_sizes = residuals
dr = fused_drhs_kernel(x, w, dy, group_sizes)
dx_norm = op(
dy,
r,
group_sizes=group_sizes,
ragged_dot_dimension_numbers=pallas_mosaic_tpu_v2.DLHS_RAGGED_DOT_DIM_NUMS,
zero_initialize=True,
)
_, rms_vjp = jax.vjp(rms_norm, x, w)
dx, dw = rms_vjp(dx_norm)
return dx, dw, dr, None
fused_gmm.defvjp(fused_gmm_fwd, fused_gmm_bwd)
Using Bare Fusible Kernels
Prefer @op_fusion.fuse with Tokamax operator wrappers (tokamax.ragged_dot
or PallasMosaicTpuV2RaggedDot) over bare fusible kernels and
@jax.experimental.pallas.fuser.fuse to maintain backward compatibility and
retain input canonicalization, autotuning, and metadata serialization. If you
invoke low-level gmm_v2 / tgmm_v2 kernels directly, use
gmm_fuser.gmm_v2_fusible or gmm_fuser.tgmm_v2_fusible inside either
@op_fusion.fuse or @jax.experimental.pallas.fuser.fuse:
from jax.experimental.pallas import fuser
from tokamax._src.ops.gmm_v2 import gmm_fuser
@jax.jit
@fuser.fuse
def fused_bare_gmm(x, w, rhs, out_bias, group_sizes):
out = gmm_fuser.gmm_v2_fusible(rms_norm(x, w), rhs, group_sizes)
return jax.nn.silu(out + out_bias)
When no prologue or epilogue ops surround gmm_v2_fusible / tgmm_v2_fusible,
trivial 0-equation fusions are unwrapped so the compiled kernel takes the exact
unfused path.
Supported Operations and Constraints
- Currently supported Tokamax ops:
PallasMosaicTpuV2RaggedDot(implementation="mosaic_tpu_v2", covering both forward/dlhsgmm_v2and drhstgmm_v2). - Supported prologue / epilogue JAX primitives: Governed by
JAX Pallas Fuser.
Commonly supported ops include pointwise unary/binary math,
select/clamp, dtype casts,broadcast_in_dim,transpose,concatenate/stack/split, andreduce_sumacross a full block axis (such asRMSNorm). Ops without fuser block-spec rules—such aslax.pad, control flow (scan/while/cond), andreduce_max/reduce_min—cannot be fused. PallasMosaicTpuV2RaggedDotfusion constraints:size_mmust be aligned to the TPU sublane tiling (size_lhs_sublane) whenlhsoroutis fused ingmm_v2(and whenlhsorrhsis fused intgmm_v2).- When
outis fused,out_size_n(andsize_kfortgmm_v2) must be aligned tonum_lanes(128) / tile size, and only single-output epilogues are supported. partial_sumis not supported with fusion.- Fused
rhsingmm_v2is not supported together withfuse_actor sub-byte packed weights.
Authoring a Fusion-Compatible Kernel and Op
Architecture Overview
Fusion dispatch is split between the public facade, the internal adapter, the
Op class, and the kernel's companion fuser module:
tokamax/ops/op_fusion.pyre-exportsfuseas the public entry point.tokamax._src.ops.fuser_adaptersets a JAX user context flag (in_op_fusion_context()) aroundjax.experimental.pallas.fuser.fuse.- When
Op.__call__runs inside that context, it checks whether the subclass overrides_fusible_fwd(raisingNotImplementedErrorif not) and delegates tofuser_adapter.call_fusible_op. call_fusible_opsuspends the fusion context while resolvingbound_args.default_config, binds afuser.fusibleprimitive over the array leaves, materializes any argument not listed inOp.fusible_argnames, and invokesOp._fusible_fwd(*args, out_fusions=out_fusions, config=config, **kwargs)insidebound_args._tokamax_metadata(config).
Step 1: Declare fusible_argnames and Override _fusible_fwd on Your Op
On your backend Op subclass (see
PallasMosaicTpuV2RaggedDot
for a full reference):
- Set
fusible_argnamesto the parameter names whose array leaves should remainfuser.Fusionclosures. All other array arguments (such as metadata, indices, orgroup_sizes) are automatically materialized to regularjax.Arrays bycall_fusible_opbefore_fusible_fwdruns. - Override
_fusible_fwd. Unlike_fwd(which returns(out, residuals)),_fusible_fwdreturns the outputTdirectly and receivesout_fusionsand the resolvedconfig. - For composite PyTree operands whose leaves should not be fused (for example,
pre-quantized
QArray(qvalue, scale)weights), materialize them explicitly withfuser_adapter.materialize(...).
from typing import ClassVar, override
import jax
from tokamax._src.ops import fuser_adapter
class MyPallasTpuOp(MyBaseOp[Config, None]):
config_cls: ClassVar[type[Config]] = Config
fusible_argnames: ClassVar[tuple[str, ...]] = ("lhs", "rhs")
@override
def _fusible_fwd(
self,
lhs: jax.Array | fuser_adapter.InputFusion,
rhs: jax.Array | fuser_adapter.InputFusion,
*,
out_fusions: fuser_adapter.OutputFusion | None = None,
config: Config,
**kwargs,
) -> jax.Array:
return my_kernel_fusible(
lhs, rhs, config=config, out_fusions=out_fusions, **kwargs
)
Step 2: Keep Fuser Integration in a Companion *_fuser.py Module
Follow Tokamax's separation between the core kernel (gmm_v2.py, tgmm_v2.py)
and its fuser integration (gmm_fuser.py):
- Keep
jax.experimental.pallas.fuserimports out of the core kernel file so the unfused kernel remains self-contained and easy to read. Unwrap 0-equation trivial
fuser.Fusionclosures before building your fusion spec:def _unwrap_trivial_input_fusion(x): if isinstance(x, fuser.Fusion) and not jax.make_jaxpr(x)().jaxpr.eqns: return x() return xfuser.fusiblewraps every input array leaf in aFusion, even when that operand has no producer ops. Unwrapping trivial closures keeps untouched operands on the unmodified unfusedBlockSpecpath with zero runtime overhead.If your base kernel is decorated with
@jax.jit, callinspect.unwrap(base_kernel)on the fusible path—@jax.jitrejectsfuser.Fusionarguments.If bare-kernel callers should also be able to invoke your fusible entrypoint directly inside
@fuser.fuse, detect barejax.Arraycalls and bindfuser.fusibleautomatically (asgmm_fuser.gmm_v2_fusibledoes).
Step 3: Unpack Fusions and Transform BlockSpecs with fuser_adapter
Use the shared helpers in
tokamax._src.ops.fuser_adapter
rather than calling low-level fuser APIs directly:
Unpack an input prologue:
lhs_fn, lhs_values, lhs_values_type = fuser_adapter.unpack_input_fusion(lhs)Returns
(fn, values, values_type)wherevaluesis the tuple of HBM arrays closed over by the prologue andvalues_typeholds theirShapeDtypeStructs.Unpack a single-output epilogue:
out_fn, out_values, out_values_type, out_init = ( fuser_adapter.unpack_output_fusion(out_fusions, raw_out_type) )Unwraps Pallas Fuser's 1-tuple output convention into a single
ShapeDtypeStruct(out_init) and raisesNotImplementedErroron multi-output epilogues.Pull an input
BlockSpecbackward through a prologue:pulled_lhs_fn, lhs_val_specs = fuser_adapter.pull_input_spec( lhs_fn, lhs_spec, lhs_values_type, grid_len=grid_len, num_scalar_prefetch=0, )Push and pull an output
BlockSpecthrough an epilogue:pulled_out_fn, out_fusion_input_specs, final_out_spec = ( fuser_adapter.push_pull_block_spec( out_fn, out_spec, out_values_type, raw_out_type, grid_len=grid_len, num_scalar_prefetch=0, ) )fuser_adapter.push_pull_block_spechandles shape-preserving reductions with size-1 broadcasts (such asRMSNormepilogues that trip rawfuser.push_block_spec), verifies that the pulled spec matches the kernel's writtenout_spec, unwraps the 1-tuple return value, and maintains compatibility across supported JAX releases.
Step 4: Wire Pulled Specs and Callbacks Into the Pallas Kernel
- Pass captured HBM inputs alongside their pulled
BlockSpecs:- Input prologue
valuesride in place of the fused operand (paired withlhs_val_specs). - Output epilogue side inputs (
out_values, such as an epilogue bias or residual) are passed as read-only HBM kernel inputs paired without_fusion_input_specssopltpu.emit_pipelineprefetches their tiles into VMEM. When unfused, pass an empty tuple()for both refs and specs (0 PyTree leaves).
- Input prologue
- Use distinct PyTree types per fusion mode:
pltpu.emit_pipelinemapsjax.tree.map(..., in_specs, in_refs), so each input ref container and itsin_speccontainer must share the exact same@jax.tree_util.register_dataclassPyTree type (for example,LhsRef[T]for unfusedlhsvs.FusedLhsRef[T]for prologue-fusedlhs).
Evaluate the pulled callbacks on VMEM tiles inside the kernel body:
- Every pulled function has signature
fn(program_ids, scalar_prefetch, *args). - Inside
pltpu.emit_pipeline(..., _explicit_indices=True), passpids = step.index(otherwisetuple(pl.program_id(i) for i in range(grid_len))) and()whennum_scalar_prefetch=0:
# Prologue: compute the input tile from prefetched VMEM refs. tiled_lhs = lhs_fn(pids, (), tuple(r[...] for r in lhs_val_refs)) # Epilogue: transform the accumulated output tile before storing. out_vals = tuple(r[...] for r in tiled_out_fusion_input_refs) out_tile = out_fn(pids, (), out_vals, acc_tile)- Every pulled function has signature
Common Pitfalls
pl.BoundedSliceandreshape: Pallas Fuser's reshape pull rule rejectspl.BoundedSlice. If your unfused kernel reshapes 2D HBM inputs to 3D sublane-tiled refs beforeemit_pipeline, convert the spec to a 2DBlockSpec((pl.BoundedSlice(tile_m), col_dim), ...)before pulling or pushing through fused operands (see_to_2d_bounded_specingmm_fuser.py).- Epilogue ordering with row masking: If your kernel masks out-of-group or
padding rows to zero and combines partial sublane tiles across steps, run
out_fnbefore applying the zero mask so non-zero epilogues (like+ bias) do not corrupt masked rows. pl.kernelwithSequenceout_type:pl.kernelunpacks atupleorlistout_typeinto multiple positionalout_refarguments. Usingfuser_adapter.unpack_output_fusionkeeps single-outputout_initas a bareShapeDtypeStruct.- Host-side padding vs. fused shapes: Because Pallas Fuser does not have a
lax.padrule or unalignedlax.slicepush/pull rules, validate alignment requirements early in your fusion spec'sprepare()method and raiseNotImplementedErrorfor unsupported shapes rather than paddingFusionobjects. - Helper
Ops inside_fusible_fwd:call_fusible_opsuspendsin_op_fusion_context()duringdefault_configresolution and inside_fusible_fwd, so calling an unfused helperOpinside_fusible_fwdruns its normal_fwdpath without triggering nested fusion errors.
Testing Checklist
When adding fusion support to an operator, cover:
- Prologue fusion (pointwise and full-axis reduction such as
RMSNorm), epilogue fusion (with and without external side inputs like bias or residual), and combined prologue + epilogue. - No-fusion parity: Verify calling the fusible entrypoint with no
surrounding ops (plain,
@op_fusion.fuse, and@fuser.fuse) produces bitwise-identical output (chex.assert_trees_all_equal) to the unfused kernel. - Explicit
@jax.custom_vjpgradients: Verify forward values and gradients match the unfused reference. - Unsupported combinations: Verify clear
NotImplementedErrormessages for unaligned shapes or incompatible kernel options.
Reference Implementation
- Public facade:
tokamax/ops/op_fusion.py Opdispatch and shared helpers:tokamax/_src/ops/fuser_adapter.py- Kernel fuser integration (
GmmFusionSpec/TgmmFusionSpec):tokamax/_src/ops/gmm_v2/gmm_fuser.py Op._fusible_fwdintegration:tokamax/_src/ops/ragged_dot/pallas_mosaic_tpu_v2.py- Kernel and
Opfusion tests:tokamax/_src/ops/gmm_v2/gmm_fuser_test.pyandtokamax/_src/ops/ragged_dot/pallas_mosaic_tpu_v2_test.py