HPC 101 - Components of a HPC

Introduction

When we refer to the UFS HPC for example, we are actually referring to the supercomputer infrastructure provided by the UFS.

A super computer is composed of several servers (referred to as nodes which are connected via a high-speed network interface and coordinated via software frameworks).

The general layout of the UFS HPC can be seen below, and will be referred to in our ongoing discussion:

Nodes

The node is basic constituent of an HPC. On the hardware level, it is a server. Thus, each node may have multiple CPUs and allot of memory. Such a node is often described as a CPU node. Nodes may also have special accelerators, such as the NVIDIA GPUs and the older Intel Phi accelerators.

On the software level, all nodes run a variant of Linux with PBS (Portable Batch System) installed to accept job requests submitted by users, and MPI and OpenMP and Open MPI to allow these jobs to be distributed on and across nodes respectively.

Most nodes are designated as compute nodes, i.e. nodes that are used to perform all the computing required of an HPC. However, single purpose nodes with other functions are also present, such as the login node in the layout above. The function of the login node is to provide an entry point to users into the HPC environment, and as such, performing compute tasks on this node is prohibited, as this would slow down access to other HPC users.

CPUs: Processors and Cores

Modern CPUs utilize multi-core architectures, meaning that a CPU contains multiple, independent processing units, referred to as physical cores. In addition to this, hyper-threading / SMT technology also allows CPU cores to be effectively "doubled", leading to nodes that report up to 64 cores (on the UFS HPC).

First, understand that this doubling is not always a true doubling, as a doubled set of cores may share memory, cache and/or other resources, which can affect performance in sometimes unforeseen ways. In this case it is often wise to request cores in multiples of two. Thus, the amount of cores reported by the HPC should in practice be seen as the amount of threads (which it technically is).

Because this nomenclature is confusing, all the documentation on the UFS HPC site will use physical cores when referring to independent processing units, and will use cores (threads) to denote threads which are reported as cores by the system.

Next, it is important to understand that not all cores are created equal. The system does NOT distinguish between the various types of cores. It only reports the amount on threads on node. For example, each Intel MIC node contains 68 physical cores (272 threads), however one of these physical cores are not equvalent in computing power and features to one of the Xeon CPU physical cores found on a CPU node.

Homogenous and Heterogeneous HPC infrastructures

In the recent past CPU-only nodes were the de facto constituents of HPC infrastructures. These infrastructures are known as homogeneous infrastructures. However, with the emergence of GPUs as more general-purpose compute accelerators and the increasing popularity of machine learning and AI, recent years have seen HPC infrastructures becoming more hetergenious in terms of their composition. Thus, modern HPC clusters often have a mixture of CPU-only, GPU and other accelerator nodes and are thus known as heterogeneous HPC infrastructures.

As the UFS HPC contains CPU-only, GPU and Intel MIC Phi nodes, the infrastructure provided by the UFS is an example of a heterogeneous HPC infrastructure.