-mpmd-move-transfers-to-producer

Moves transfers next to their producers.

Moves transfers next to their producers: if the operand is a block argument, move the transfer to the beginning of the block, otherwise move it after the defining op.

-mpmd-pipeline-scheduler

Reorders the fragments to obtain a given pipeline schedule.

Reorders fragments according to a pipeline schedule. The scheduling algorithm relies on a happens before function that takes two fragments f1 and f2 as arguments and checks if f1 must be scheduled before f2. This function requires that:

• f1 and f2 are assigned to the same mesh,
• f1 and f2 are both scheduling units (i.e., user defined fragments with call_counters defined), and
• f1 does not depend on f2 and f2 does not depend on f1. And it should be enough to express many state-of-the-art pipeline schedules.

For each fragment f1 that must be scheduled before f2, the scheduler pass creates a control-dependency from f1 to f2. Then, it applies a topological sort on the module to guarantee that all dependencies are respected (and the program is in a valid SSA form). Finally, the pass removes from the graph any control-dependency introduced.

Options

-must-happen-before : A comparator that determines whether a fragment must be scheduled before another. Can be parsed from a built-in `PipelineSchedule` as follows: `builtin:<schedule-as-string>`.

-mpmd-remat-fragment

Rematerializes fragments.

Finds pairs of fragments (forward+backward) that need to be rematerialized and clones every forward fragment before its backward user, replace all backward uses of values produced by the forward fragment with the cloned counterparts. This can be used for activation rematerialization in pipeline parallelism.

When merge_remat_fragments is true, then we merge the remat fragments into their consumer fragments.

Options

-merge-remat-fragments : Whether to merge the remat fragments into their consumer fragments.

-mpmd-scheduling-units-verifier

Verifies if the program contains the required scheduling units.