Performance Evaluation of Pseudospectral Ultrasound Simulations on a Cluster of Xeon Phi Accelerators

conference paper

English

The ability to perform large-scale ultrasound simulations has generated significant interest in medical ultrasonics, including for treatment planning in therapeutic ultrasound, and image reconstruction in photoacoustic tomography. However, routine execution of such simulations using traditional computational methods, e.g., finite difference time domain, is considered intractable due to the computational and memory requirements. The k-Wave toolbox alleviates these requirements by employing a k-space corrected pseudospectral method. This significantly reduces the spatial and temporal grid resolution, however, at the cost of introducing global all-to-all communication through the use of the fast Fourier transform. To improve data locality, reduce data movements and allow efficient use of accelerators, we recently implemented a domain decomposition technique based on local Fourier basis. Nowadays, the trend in parallel computing is towards the use of accelerated nodes where computationally intensive tasks are offloaded from processors to accelerators. In this paper, we investigate the performance aspects of the distributed k-Wave implementation running on the Salomon cluster equipped with 864 Intel Xeon Phi (Knight's Corner) accelerators. The paper shows that running large simulations across many accelerators is not a trivial task. The main obstacle is the instability of Intel MPI on Infiniband interconnection once the number of accelerators exceeds 32. Beyond this limit, the use of service 1Gbps interconnection is the only solution. The second problem is low performance of the fast Fourier transforms ranging from 1% to 50% of a single 12-core CPU. Finally, there is no support for fast LustreFS file system. Despite these factors, the observed strong and weak scaling is comparable with a cluster of CPU, but the absolute running time is between 4.3x and 1.8x longer in the worst and best case, respectively. However, the accounting policy for Salomon's accelerators is much more favorable and thus their employment reduces the computational cost significantly. 

Ultrasound simulations, Local Fourier basis decomposition, Pseudospectral methods, Ultrasound, k-Wave toolbox, Intel Xeon Phi, Knight's Corner, MKL, MPI, OpenMP

VAVERKA, F.; TREEBY, B.; JAROŠ, J.

8. 1. 2021

Springer Nature Switzerland AG

Cham

978-3-030-67076-4

High Performance Computing in Science and Engineering. HPCSE 2019

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

99

115

17

https://link.springer.com/chapter/10.1007/978-3-030-67077-1_6

HPCSE_Vaverka_2021.pdf