Results of HPC executions can be affected by two types of errors: uncontrolled ones (silent data corruption) and controlled ones (lossy compression). This project investigates the detection and impact of these different errors and the challenge raised by their potential concurrent occurrence.

Silent data corruption: Computer circuitry continues to become denser, as gate sizes continue to shrink, and the number of layers continues to grow. With denser circuits, it is increasingly harder to prevent transient hardware errors. Furthermore, the power consumption of circuits and their manufacturing costs can be reduced if more frequent errors can be tolerated.

The standard practice for error recovery in HPC is to store periodic checkpoints and the restart from the last checkpoint if an error is detected. It is critical that errors be detected before a corrupted checkpoint is stored. This motivates our research: Can silent errors (i.e., hardware errors not detected by hardware) be detected by software?

The approach we took is to focus on single bit flips (reasonable, if errors are rare) and on iterative numerical algorithms. A bit flip may have a negligible impact on the computation. If it is significant, then we expect to see in the simulation a point perturbance that propagates over successive iterations. We consider a correct simulation as “noise” and try to detect the superimposed “signal” of the perturbation caused by the bit flip. We do so using machine learning: A Convolutional Neural network (CNN) is trained to distinguish normal simulations from perturbed ones.

We tested our approach using various configurations of the FLASH code that is widely used for magneto-hydrodynamic simulations. We achieve almost perfect recall, with a false positive rate between 0 and 2.4%, depending on configuration, if the classifier is run at the end of the iteration when the error occurred. Furthermore, a high recall rate is maintained, in many cases, even if the diagnostic is run up to 10 iterations after the error occurred.

Lossy compression: Data reduction is becoming a necessity for many numerical simulations that generate more data than can be stored, communicated, and analyzed. Error bounded lossy compression provides a reliable way to reduce scientific datasets while respecting user requirements regarding point wise accuracy. To be usable, lossy compression needs to be fast and users must be able to understand its impacts not only on data points, but also globally using statistics and on users’ analysis.

To increase the speed of lossy compression, we optimized cuSZ, a cuda version of SZ and we optimized the Huffman coding specifically for GPUs. The new version (cuSZ+) improves the compression throughputs and ratios by up to 18.4× and 5.3×, respectively, over CUSZ for seven real-world HPC application datasets on NVIDIA V100 and A100 GPUs. Concerning Huffman coding we parallelized the entire algorithm, including codebook construction and we proposed a novel reduction based encoding scheme that can efficiently merge the codewords on GPUs. Experiments show that our solution can improve the encoding throughput by up to 6.8X on NVIDIA V100 over the state-of-the-art GPU Huffman encoder.

Ultimately, users’ trust in lossy compression relies on the preservation of science: same conclusions should be drawn from computations or analysis done from lossy compressed data. In order to better understand the impact of lossy compression error on scientific data and on post-hoc analysis, we analyzed 8 different use-cases and studied 3 different levels of error analysis: visualization  (L1), quantitative error analysis (L2) and quantitative analysis of user analysis deviation (L3). Experience from scientific simulations, and instruments show that these three levels of analysis are necessary to quickly detect unacceptable compression generated artifacts, assess the profound nature of the compression error and analyze quantitively the impact of compression error on user analysis. These 3 levels were used to tune and improve lossy compression pipelines in these 8 use-cases to fulfil users accuracy requirements on data and post-hoc analysis. 

SDC mitigation/Compression interplay: To detect and correct SDC happening during lossy compression, we designed and implemented the first algorithm-based fault tolerance (ABFT) algorithm for lossy compression. Our solution incurs negligible execution overhead in the fault-free situation. Should soft errors occur, it ensures decompressed data is strictly bounded within user's requirement, with a very limited degradation of compression ratio and low overhead.

Last Modified: 11/15/2021
Modified by: Marc Snir