CoreWeave Fabric Diagnostics

> Community-contributed. Not affiliated with, endorsed by, or sponsored by

> CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Detects when a CoreWeave multi-node GPU job has silently fallen off the InfiniBand fabric

onto TCP — the failure that makes distributed training run at a fraction of the hardware's

speed while every GPU keeps billing at the full rate — and gives the exact fix.

Overview

On a CoreWeave multi-node job, NCCL should carry collectives over **InfiniBand with

GPUDirect RDMA (NET/IB). If any one of three conditions is missing, NCCL silently

falls back to TCP sockets** (NET/Socket): the job still runs, still converges, and

raises no error — but all-reduce throughput collapses (commonly cited as **5-20x

slower** [[nccl]]) because it now crosses the Ethernet control plane instead of the

400 Gb/s-class fabric. You keep paying full GPU rate for a multi-node run that performs

like a badly-connected one. This is the single highest-dollar invisible failure on the

platform, and nothing in the default output flags it.

The diagnosis is deterministic: the bundled scripts/fabric-check.py greps the pasted

NCCL_DEBUG=INFO log for the decisive Using network line (and NET/IB vs NET/Socket),

parses the pod-spec's resources block for the RDMA device request, reads ibstat port

state, and echoes any allreduceperf bus bandwidth — then emits a VERDICT with the

rdma_engaged/transport call, the missing conditions, and the fix. The LLM never

eyeballs which transport is in use; the script decides. Deep grounding lives in

references/, loaded only when a leg of the diagnosis needs it.

Prerequisites

• An NCCL_DEBUG=INFO log from the actual run — the primary signal. Re-run the job

(or one rank) with NCCL_DEBUG=INFO set and capture stderr. The decisive line is

Using network IB (good) vs Using network Socket (the fallback). This is the one input

the skill really needs; everything else corroborates.

• Optional, for a full diagnosis: the pod/job spec (kubectl get pod NAME -o yaml) to

check the RDMA device request; ibstat output from the node for port health; and

allreduceperf results from CoreWeave's [nccl-tests][nt] to measure bus bandwidth.

• python3 to run the deterministic checker (stdlib only).
• kubectl (read-only) if corroborating the live pod spec / node cordon state.

Authentication. Nothing secret is read. If the pod spec is pulled live, kubectl uses

the existing $KUBECONFIG; the skill only ever runs kubectl get (read-only) — it never

cordons, drains, or applies.

The three required conditions (all must hold, or NCCL falls back)

1. The RDMA device is requested in BOTH resources.requests AND resources.limits

(rdma/ib: 1). If it is in only one — or absent — the device plugin does not inject the

IB device into the pod and NCCL never sees a HCA. [unverified — the exact resource key

(e.g. rdma/ib) depends on the installed RDMA device-plugin config; confirm with

kubectl describe node / kubectl get node -o yaml.]

1. NCCLIBHCA=ibp and NCCLSOCKETIFNAME=eth0 are set (CoreWeave's documented

values [[cw]]) — unless you launch via the MPI Operator, which manages this network

config for you [[nt]].

1. NCCL_DEBUG=INFO then confirms NET/IB (ideally a GPU Direct RDMA Enabled line).

If it shows NET/Socket / Using network Socket, RDMA is not engaged.

Full checklist with verification commands: references/rdma-engagement-checklist.md.

Instructions

The pipeline is gather → verdict → fix → confirm. The script does the transport call;

references/ carry the grounding:

1. Gather the NCCLDEBUG=INFO log (required) plus any pod-spec / ibstat / allreduce_perf

output you have. Concatenate them into one paste — the checker keys on each signal

independently.

1. Run the deterministic checker to get the VERDICT.
2. If the verdict is fallback (Socket), apply the three-condition fix and re-run.
3. If the verdict is IB but degraded, chase the degraded signal (port down / low busbw).
4. On NVSwitch systems with a stuck fabric, use the Fabric Manager reset order.

Step 1: Gather the evidence

The log is the load-bearing input. If the user has not run with NCCL_DEBUG=INFO, tell

them to — without it, transport selection is unknowable. To pull the live pod spec:

kubectl get pod "$POD" -o yaml > pod.yaml

Step 2: Run the deterministic checker (it makes the call, not the model)

Pipe everything you gathered to fabric-check.py. It greps for the decisive `Using

network line, the resources block, ibstat state, and any Avg bus bandwidth`:

cat nccl-debug.log pod.yaml ibstat.txt allreduce.txt 2>/dev/null | \
  python3 scripts/fabric-check.py

The verdict names rdma_engaged (yes/no/partial/unknown), the transport in use, the

missing conditions, and the fix. Use --json to capture the structured result for further

processing. Reading the log by eye is what this step exists to prevent — see

references/nccl-debug-reading.md for what each line

means.

Use Glob to gather multiple pasted log files when a run spans several ranks, Write the

verdict report to the working directory, and Edit it to refine the fix as the user

iterates on the manifest.

Step 3: If the verdict is fallback (NET/Socket) — apply the three-condition fix

This is the money case. Fix in order (the checker prints the same list):

1. Add rdma/ib: 1 to both resources.requests and resources.limits.
2. Set NCCLIBHCA=ibp and NCCLSOCKETIFNAME=eth0 (or launch via the MPI Operator).
3. Re-run with NCCL_DEBUG=INFO and confirm the log now shows NET/IB +

GPU Direct RDMA Enabled, not NET/Socket.

If the log shows NCCLIBDISABLE=1, that alone forces sockets — set it to 0 (RoCE and

IB both need the IB verbs transport enabled [[env]]).

Step 4: If the verdict is IB-but-degraded — chase the degraded signal

RDMA can be engaged yet slow. Two corroborating checks:

• ibstat — every port must read State: Active / Physical state: LinkUp. A port

Down/Polling, or a link that flaps, drags the whole collective; CoreWeave

auto-cordons flapping links, so a shrinking node count mid-run is a fabric symptom.

• allreduceperf bus bandwidth — compare the reported busbw against **CoreWeave's

published nccl-tests manifest baseline for your GPU count + NCCL version** [[nt]]. Do

not compare against a fixed number: the baseline moves with GPU type, node count, NCCL

version, and SHARP. The checker echoes the observed figure tagged `[unverified vs

baseline]` precisely so nobody reads it as a hard pass/fail.

Details + the busbw-vs-algbw distinction: references/allreduce-baseline.md.

Step 5: NVSwitch systems — Fabric Manager reset order

On NVSwitch/NVLink systems, a wedged fabric shows up as NVLink/NVSwitch errors rather than

IB fallback. The safe reset order is **stop Fabric Manager → reset the GPUs → start Fabric

Manager**, never the reverse:

sudo systemctl stop nvidia-fabricmanager
sudo nvidia-smi -r            # GPU reset
sudo systemctl start nvidia-fabricmanager

`[unverified — service unit name and reset support vary by image/driver; on managed

CoreWeave nodes prefer opening a support ticket / cordoning over an in-place reset.]`

Output

• A VERDICT line stating whether RDMA is engaged, the transport actually in use, and —

for the fallback case — the plain-language cost framing (running on TCP, paying full GPU

rate for a fraction of the throughput).

• The missing-conditions list — which of the three required conditions is absent, each

one sufficient on its own to force the fallback.

• The ordered fix — the rdma/ib-in-requests-AND-limits change, the env vars, and the

re-verify step.

• Degraded-fabric signals when RDMA is engaged but slow — down/flapping IB ports and the

observed busbw (tagged [unverified vs baseline]).

Error Handling

Error	Cause	Solution
Verdict is unknown	No NET/IB / NET/Socket / Using network line in the paste	Re-run the job with NCCL_DEBUG=INFO and capture stderr; without it transport is unknowable.
Verdict Socket but the pod "has RDMA"	rdma/ib in limits only (or only requests)	Add it to BOTH blocks; the device plugin injects the IB device only when the resource is requested.
NET/IB present yet training still slow	GDR not actually enabled; nvidia-peermem unloaded → traffic stages through host memory	Confirm a GPU Direct RDMA Enabled line; verify nvidia-peermem is loaded on the node [[nccl]].
busbw "looks low"	Compared against a wrong/guessed baseline	Compare only against CoreWeave's nccl-tests manifest baseline for your GPU count + NCCL version; the number is workload/version-dependent.
Nodes drop out mid-run	Flapping IB link → CoreWeave auto-cordon	Check ibstat for Physical state != LinkUp; the cordoned node's link is the cause, not your job.
rdma/ib resource not schedulable	Wrong resource key for the installed device plugin	Confirm the exact key with kubectl describe node (search the Allocatable list) and substitute it.

Examples

Example 1: "Our 4-node H100 training run got slow — is RDMA even working?"

The user pastes an NCCL_DEBUG=INFO excerpt plus the pod spec. The checker finds `Using

network Socket and rdma/ib only in requests`, and verdicts:

### VERDICT: RDMA is NOT engaged -- NCCL fell back to TCP (NET/Socket). Multi-node collectives are running over the Ethernet control plane, commonly 5-20x slower for the same GPU-hours -- you pay full GPU rate for a fraction of the throughput, and NCCL raised no error.

- RDMA engaged: **no**
- Transport in use: **Socket**
- Missing conditions (each one alone forces a silent TCP fallback):
    - `rdma/ib` missing from resources.limits
    - `NCCL_IB_HCA` not set (e.g. `ibp`) -- unless the MPI Operator manages it

**The fix (in order):**
1. Request the RDMA device in BOTH requests AND limits: `rdma/ib: 1` (if it is in only one, the device plugin will not inject the IB device).
2. Set `NCCL_IB_HCA=ibp` and `NCCL_SOCKET_IFNAME=eth0` (CoreWeave values), or let the MPI Operator manage them.
3. Re-run with `NCCL_DEBUG=INFO` and confirm the log now shows `NET/IB` and `GPU Direct RDMA Enabled` -- not `NET/Socket` / `Using network Socket`.
4. Confirm each IB port is `State: Active` / `Physical state: LinkUp` via `ibstat`; a flapping link gets auto-cordoned by CoreWeave.

Example 2: "RDMA is on but all-reduce bandwidth seems low"

The log shows NET/IB and GPU Direct RDMA Enabled, so the checker returns

rdma_engaged: yes. It then surfaces the ibstat port that reads `Physical state:

Polling as a degraded signal and echoes the observed busbw tagged [unverified vs

baseline]`, directing the user to compare against CoreWeave's nccl-tests manifest for their

GPU count + NCCL version rather than a guessed number.

Resources

• references/rdma-engagement-checklist.md — the three required conditions + how to verify each, cited.
• references/nccl-debug-reading.md — reading NCCL_DEBUG=INFO: NET/IB vs NET/Socket, the decisive Using network line, GDR.
• references/allreduce-baseline.md — allreduceperf busbw/algbw and why the baseline is never hardcoded.
• Sibling: coreweave-gpu-cost-leak-hunter dollarizes idle/right-sizing spend; this skill finds the throughput leak (fabric fallback) that a cost report cannot see.

[cw]: https://docs.coreweave.com/docs/products/networking/hpc-interconnect/use-gpudirect-rdma

[nt]: https://github.com/coreweave/nccl-tests

[nccl]: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html

[env]: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html