Minder

Minder: Faulty Machine Detection for Large-scale Distributed Model Training link

Challenges to solve

1. Any machine could fail in various ways
2. The normal state of a monitoring metric is task-dependent
3. The correlation between fault types and monitoring metrics is not necessarily one-to-one
4. Noises exist in time series monitoring data

Solution proposed by Minder

1. Machine level similarity: For challenge 1 and 2, if a machine undergoes a fault, its monitoring data will display distinctive differences, offering an opportunity for detection.
2. Machine level continuity: For challenge 2, most abnormal patterns last for over 5min. if we recognize a machine displaying such dissimilarity continuously for a period, the machine may be faulty.
3. Individual Learning-Based Denoising Models for Each Monitoring Metric:
   • For challenge 4, variational autoencoders can learn embedding schemes that can infer the generation factors for most of the training data.

Metrics to monitor (bold are used in detection, others are collected but not used):

1. CPU Usage: Percentage of CPU time being used.
2. PFC Tx Packet Rate: Periodic counts of PFC packets sent by RDMA-enabled devices. - RoCE only?
3. GPU Duty Cycle: Percentage of time over the past sample period when the accelerator is active.
4. GPU Power Draw: Periodic counts of the GPU power consumption.
5. GPU Tensor Core Activity: Percentage of cycles when the tensor (HMMA / IMMA) pipe is active.
6. GPU Graphics Engine Activity: Percentage of time when any portion of the graphics or compute engines are active.
7. GPU NVLink Bandwidth: The rate of data transmitted/received over an NVLink.
8. Memory Usage: Percentage of memory being used.
9. Disk Usage: Percentage of storage space being used on a disk.
10. TCP Throughput: Periodic counts of the amount of TCP data being transmitted by a NIC.
11. TCP+RDMA Throughput: Periodic counts of the amount of TCP and RDMA data being transmitted by an NIC.
12. GPU Memory Used: The amount of GPU memory being used by processes.
13. GPU Temperature: The temperature of a GPU while it is operating, measured in degrees Celsius.
14. GPU SM Activity: Averaged percentage of time when at least one warp is active on a multiprocessor.
15. GPU Clocks: The clock speed of a GPU, reflecting the frequency of the GPU’s processor.
16. GPU FP Engine Activity: Percentage of cycles when the FP pipe is active.
17. GPU Memory Bandwidth Utilization: Percentage of cycles when data is sent to or received from the device memory.
18. PCIe Bandwidth: The rate of data transmitted/received over the PCIe bus.
19. PCIe Usage: Percentage of the bandwidth being used on the PCIe bus.
20. ECN Packet Rate: Periodic counts of ECN packets transmitted/received by a NIC.
21. CNP Packet Rate: Periodic counts of CNP packets transmitted/received by a NIC.

Analytics:

1. Metrics data are grouped into a time window.
2. Within the time window, align sample points across all sampled machines. If sample data points are missed, use data from nearest sampling time for padding.

3. Normalize data points based on upper and lower limits of each metric with min-max normalization.

   \[x' = \frac{x - min}{max - min}\]
4. Fed the normalized data to the corresponding metric LSTM-VAE model to get the embedding.
5. Calculate pairwise Euclidean distances of embeddings between every two machines.
6. For each machine, get the sum of the distances to other machines

7. Calculate the normal/z-score of the sum of the distances of each machine. avg(x) is the average of x and std(x) is the standard deviation of x.

   \[z =\frac{x - avg(x)}{std(x)}\]
8. The machine with the maximum normal score is probably the faulty one. If the maximum normal score is higher than a similarity threshold, the machine is assumed as a candidate of the time window
9. Continuity check: shifts the time window with one data sample to detect the potentially faulty machine for new windows. If the same machine is detected with consecutive times that exceed a continuity threshold, 4mins, it is considered a truly faulty machine.