Publication detail

Performance Evaluation of Pseudospectral Ultrasound Simulations on a Cluster of Xeon Phi Accelerators

VAVERKA, F. TREEBY, B. JAROŠ, J.

Original Title

Performance Evaluation of Pseudospectral Ultrasound Simulations on a Cluster of Xeon Phi Accelerators

Type

conference paper

Language

English

Original Abstract

The ability to perform large-scale ultrasound simulations has generated significant interest in medical ultrasonics, including for treatment planning in therapeutic ultrasound, and image reconstruction in photoacoustic tomography. However, routine execution of such simulations using traditional computational methods, e.g., finite difference time domain, is considered intractable due to the computational and memory requirements. The k-Wave toolbox alleviates these requirements by employing a k-space corrected pseudospectral method. This significantly reduces the spatial and temporal grid resolution, however, at the cost of introducing global all-to-all communication through the use of the fast Fourier transform. To improve data locality, reduce data movements and allow efficient use of accelerators, we recently implemented a domain decomposition technique based on local Fourier basis. Nowadays, the trend in parallel computing is towards the use of accelerated nodes where computationally intensive tasks are offloaded from processors to accelerators. In this paper, we investigate the performance aspects of the distributed k-Wave implementation running on the Salomon cluster equipped with 864 Intel Xeon Phi (Knight's Corner) accelerators. The paper shows that running large simulations across many accelerators is not a trivial task. The main obstacle is the instability of Intel MPI on Infiniband interconnection once the number of accelerators exceeds 32. Beyond this limit, the use of service 1Gbps interconnection is the only solution. The second problem is low performance of the fast Fourier transforms ranging from 1% to 50% of a single 12-core CPU. Finally, there is no support for fast LustreFS file system. Despite these factors, the observed strong and weak scaling is comparable with a cluster of CPU, but the absolute running time is between 4.3x and 1.8x longer in the worst and best case, respectively. However, the accounting policy for Salomon's accelerators is much more favorable and thus their employment reduces the computational cost significantly. 

Keywords

Ultrasound simulations, Local Fourier basis decomposition, Pseudospectral methods, Ultrasound, k-Wave toolbox, Intel Xeon Phi, Knight's Corner, MKL, MPI, OpenMP

Authors

VAVERKA, F.; TREEBY, B.; JAROŠ, J.

Released

8. 1. 2021

Publisher

Springer Nature Switzerland AG

Location

Cham

ISBN

978-3-030-67076-4

Book

High Performance Computing in Science and Engineering. HPCSE 2019

Edition

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Pages from

99

Pages to

115

Pages count

17

URL

https://link.springer.com/chapter/10.1007/978-3-030-67077-1_6

BibTex

@inproceedings{BUT168468,
  author="Filip {Vaverka} and Bradley {Treeby} and Jiří {Jaroš}",
  title="Performance Evaluation of Pseudospectral Ultrasound Simulations on a Cluster of Xeon Phi Accelerators",
  booktitle="High Performance Computing in Science and Engineering. HPCSE 2019",
  year="2021",
  series="Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
  pages="99--115",
  publisher="Springer Nature Switzerland AG",
  address="Cham",
  doi="10.1007/978-3-030-67077-1\{_}6",
  isbn="978-3-030-67076-4",
  url="https://link.springer.com/chapter/10.1007/978-3-030-67077-1_6"
}