[ci]: introduce three-layer workflows and qualify GPU batches

[richardoo-707]

What does this PR do?

Introduce the shared CI configuration and batch runner needed to separate testing into three resource tiers, while retaining existing release, documentation, example, notification, and repository-maintenance workflows during rollout.

• Layer 1: formatting and genuinely CPU-only tests on GitHub-hosted runners for every PR.
• Layer 2: fast core GPU batches 2 and 5 for trusted same-repository PRs and manual dispatch; unreviewed fork code is not executed on persistent self-hosted runners.
• Layer 3: weekly/manual full GPU batch coverage on the dedicated H20 runner, currently qualified with Torch 2.13 / CUDA 13.0. Additional runtime profiles are not provisioned by this PR.
• Shared execution: pinned source archive, centralized batch membership, fail-closed occupancy checks, per-host locking, per-batch GPU selection, process-group timeout cleanup, JUnit artifacts, and run summaries.
• An optional capacity-aware hourly workflow admits Layer 2 with two idle GPUs and Layer 3 with eight. Scheduled workflows take effect on the default branch; legacy required checks are not retired in this PR.

Also includes the runtime/test compatibility fixes discovered while qualifying existing GPU suites: checkpoint and FP8 compatibility, hermetic offline test fixtures, Llama inference tensor-parallel dimensions and cache handling, Mixtral attention compatibility, Qwen2/Qwen3 pipeline/sequence-parallel masks, and failure/worker cleanup. These are production-code changes as well as CI changes and warrant review of the affected model and checkpoint paths.

Validation and review status

Current PR head: 72d4aab337639e58a03ebee8b951e32a9b189e7a.
This includes the merge of main at df010702 in f7762b59, followed by pre-commit.ci formatting changes. The documentation conflict was resolved by retaining both the layered rollout and the upstream fork-safe checks. The PR has no merge conflicts, but GitHub currently reports the merge state as BLOCKED because acceptance/check requirements remain outstanding.

Latest post-merge Layer 3 validation

Successful run on the current PR head, completed on 2026-10-11 at 15:26 Asia/Singapore, approximately 91 minutes overall.

• fast_mode=false; the selected batches retain their original test configurations.
• All 14 requested batch commands completed successfully: 2 3 4a 4b 4c 4d 4e 5 6 7 8 9 9q 11.
• JUnit totals: 2,063 passed, 391 skipped, 0 failures, 0 errors (2,454 reported tests).
• Original one-, two-, and four-GPU configurations ran on idle devices without using the occupied GPUs 0 and 1. Eight-GPU batch 10 was intentionally not selected.
• The Qwen2 pytest item runs its internally parameterized model/configuration matrix; one reported item does not mean one configuration.

Coverage	Current-head result
Single-GPU core, Adam kernels, FP8, inference CUDA/Triton kernels	No failures in selected cases; skips remain
Two-GPU core batch 5	5 passed, 0 skipped
Four-GPU distributed foundations, FP8 collectives, training, models/inference, Qwen2	No failures in selected cases; skips remain
SmoothQuant batch 4d	0 passed, 4 skipped; functionality NOT validated
Apex-dependent Gemini batch 11	0 passed, 2 skipped; functionality NOT validated
Eight-GPU batch 10	Pending rerun on the current head

Previous real eight-GPU validation

The original full run at 90960558 had 11 successful batch commands and failures in batches 10, 8, 9, and 9q:
Previous full run.

Batch 10 initially failed tensor-parallel inference, speculative decoding, and Mixtral eight-GPU tests. After the compatibility fixes, the real eight-GPU rerun at 641cda4d succeeded using physical GPUs 0-7: 21 passed, 3 skipped, 0 failures/errors, approximately 29 minutes for batch 10. The same run also passed batches 8, 9, 9q, and 2 (156 passed and 31 skipped total).

These historical eight-GPU results are evidence for review, not a current-head eight-GPU acceptance result. The subsequent main merge did not directly change those eight-GPU tests or their core model/inference fixes, but the integrated current head still needs its own batch-10 rerun.

Additional verification

• Post-merge Linux verification: 19 CI script unit tests and 4 example-check unit tests passed.
• Runner-policy validation, Bash syntax, new CI workflow static checks, and the resolved documentation's pre-commit checks passed.
• Both current-head PR CPU test jobs passed.
• Legacy workflow static lint still reports pre-existing deprecated Action/command issues; this PR does not retire those workflows.

Remaining acceptance checks / limitations

• Rerun batch 10 with eight genuinely idle GPUs at the final PR head. Latest direct SSH inspection on 2026-10-11 at 21:44 Asia/Singapore found six idle GPUs (2-7), with GPUs 0 and 1 occupied; no eight-GPU run was launched.
• Resolve the current-head formatting/lint failures (Formatting, Lint and formatting, and pre-commit.ci). Logs show autoflake changes around Apex availability-probe imports; the full formatting checks are NOT green.
• Resolve/refresh outstanding PR GPU checks. The automatic Layer 2 run failed because no authorized idle GPUs were available at its launch; batches 2 and 5 subsequently passed in the manual Layer 3 run above, but that does not automatically turn the separate PR check green. The internal gpu-smoke check is also still marked failed; legacy build/example runs were cancelled.
• Maintainer CR and required branch-protection checks before merge.

Skipped tests are not counted as passing. The 391 current-run skips include optional dependencies (Apex, TensorNVMe, torch_int, vLLM), missing model/reference assets, unsupported or disabled kernel/test configurations, memory constraints, and missing async-test plugin coverage. In particular, batch-command success for 4d and 11 does not establish SmoothQuant or Apex functionality. The successful batch-10 run also retained three skips.

This PR is available for code review, but it is not presented as a zero-skip, all-checks-green, complete latest-head GPU acceptance. Do not retire legacy checks or migrate branch protection until rollout validation and the resource/trust policy are accepted.

Checklist

• Title follows the repository's [ci]: ... format.
• Added CI-runner unit tests and runtime-compatibility regressions.
• Documented workflow responsibilities, rollout configuration, and targeted validation.
• Merged main and resolved conflicts.
• Revalidated all 14 non-eight-GPU batches at the current PR head, with skip limitations reported.
• Recorded successful historical real eight-GPU validation.
• Linked tracking issue (none supplied).
• Final-head real eight-GPU rerun.
• Required PR checks green and maintainer approval.