CINECA · Bologna, IT
Tier-1 supercomputer · 4× NVIDIA V100 per node · ~32 PFLOPS
Marconi100 — an NVIDIA V100 node drifting toward 2× its power.
A real degradation event on CINECA's Marconi100, replayed across 30 months of continuous telemetry. OrionLinks flagged the failing node months before it reached its worst operating point — while a conventional threshold dashboard showed nothing wrong.
54 dayslead time · first signal → peak power
2.07×baseline power the node climbed to
4 / 847nodes flagged across the fleet
933 dayscontinuous telemetry analysed
The model walked Node 978 from STABLE → WATCH → DRIFTING → DEGRADING as its power draw pulled steadily away from the pattern that node had held for months under the same workload. That sustained, escalating departure from its own baseline is the signal that separates a hardware fault from ordinary day-to-day variation. Built on 698,461 daily power records.
GWDG · Göttingen, DE
A100 GPU fleet · DCGM telemetry
GWDG — every NVIDIA A100 GPU failure with telemetry, caught early.
The same model, the same thresholds, pointed at a different cluster — GWDG's A100 GPU fleet — with no re-tuning. Scored forward-only against a year of operator-logged incidents as ground truth.
16 / 16telemetered GPU failures caught
4 daysmedian lead over the operator's report
15 / 16flagged before the human incident report
69incidents in the ground-truth catalog
Three forward-only layers — self-normalised behaviour, slow-drift slope, and DCGM hardware faults — score each node on telemetry alone. The detector ported straight from Marconi100 generalised to a new vendor and workload with zero domain tuning, confirming the approach isn't cluster-specific.