Technical Details
This page describes in detail the components and operation of the compute cluster at ACCRE. For researchers requiring text for grant proposals and publications, we also provide a less technical Summary of the ACCRE Facility.
The compute cluster currently consists of more than 600 Linux systems with eight to sixteen processor cores each. Each node has at least 24 GB of memory (most have at least 128 GB) and dual Gigabit copper ethernet ports.
By the numbers
- Last updated January 2024.
| Number | Description |
|---|---|
| 3,100 | Number of researchers |
| 50 | Number of campus departments and centers |
| 5 | Number of schools |
| 737 | Number of compute nodes |
| ~15,960 | Total number of cores |
| 24-1028GB | Memory available per node |
| ~4.8PB | Disk space available through shared cluster storage |
| 22PB | Disk space available through LStore |
Details of the Cluster Design
The design reflects significant input from the investigators who use it in their research and education programs. Factors such as the number of nodes, the amount of memory per node, and the amount of disk space for storing user data is determined by demand. A schematic diagram of the system is shown below followed by a glossary of terms used in a subsequent description of the cluster:
- Bandwidth
- The rate at which data can be transferred over the network. Typically this is measured in Megabits/sec (Mbps) or Gigabits/sec (Gbps).
- Compute Node
- A node whose purpose is to run user applications. Users will not have direct access to these machines. Access will be granted solely through the job scheduler.
- Disk Server
- A machine that is used for data storage. Users will not normally have access to these machines. For more information see the description of GPFS below.
- Gateway or Management Node
- Computer designed for interactive login use. Users log in to these machines for compiling, editing, and debugging of their programs and to submit jobs.
- Gigabit Ethernet
- Commodity networking typically found in servers and has a bandwidth of 1 Gbps and latency of up to 150 microseconds in Linux. Similarly, 10 gigabit ethernet is high performance networking with a bandwidth of 10 Gbps.
- Latency
- The amount of time required to “package” data for sending over the network.
Cluster Connectivity
The ACCRE cluster has a flexible network topology which can accommodate different users and their needs while leaving room for expansion. There are two separate functional networks: (1) external network for connectivity to the outside world, and (2) a private internal cluster network.
The gateways are connected to the Internet through the external network comprised of gigabit ethernet links to one of our core 10 gigabit or 100 gigabit switches. Vanderbilt University has a 100 gigabit connection to the Internet via ACCRE. The internal cluster network is used for both file system traffic (ACCRE GPFS, LStore,) and compute job network traffic. Regular compute nodes each have a gigabit connection to the internal cluster network and only that network for security though they can initial bidirection connectivity outside ACCRE via two 10 gigabit NAT systems. Some nodes, such as Big Data nodes, have 10 or 25 gigabit internal network connections. The internal cluster network has a core with spine-leaf architecture composed of many 40 and 100 gigabit links for bandwidth scalability and fault tolerance plus classical fat tree design outside the core so nodes near each other do not compete for bandwidth. Only a constant incremental cost per machine is incurred for each additional machine.
Node installation, maintenance updates, and health monitoring
Initial OS installation is accomplished using Kickstart and CFEngine. We make an image of the configured and operational system, subsequently replicating this image across all nodes. Multiple images for different compute node configurations are stored on an infrastructure node. By using this technique it becomes possible to perform a wipe and re-install of the entire cluster in a short period of time. Similarly, Kickstart and CFEngine are also used to update a node, transferring only the data needed for the update instead of the entire image. Because of this intelligent update, most common maintenance operations, such as updating the kernel, take mere minutes.
The health of the compute nodes are monitored through the use of the open source package Nagios which supports distributed monitoring. Distributed monitoring allows data from individual management nodes to be collected on a single node. Nagios uses a client-server architecture and is written in the C programming language.
Compute cluster
Under construction
For the GPU nodes information please see this link: GPUs at ACCRE
Cluster File System
Under construction.
Data Storage and Backup
The True incremental Backup System (TiBS) is used to backup the ACCRE cluster home directories nightly using a Spectra T950 tape library. A big advantage of TiBS is that it minimizes the time (and network resources) required for backups, even full backups. After the initial full backup, TiBS only takes incremental backups from the client. To create full backups, an incremental backup is taken from the client. Then, on the server side, all incremental since the last full backup are merged into the previous full backup to create a new full backup. This takes the load off the client machine and network. The integrity of the previous full backup is also verified. (TiBS is available for all current operating systems and apart from the cluster, ACCRE also offers backup services for data located remotely. This service is through special arrangement.
Resource Allocation
A central issue in sharing a resource, such as the cluster, is making sure that each group is able to receive their fairshare if they are regularly submitting jobs to the cluster, that groups do not interfere with the work of other groups, and that research at Vanderbilt University is optimized by not wasting compute cycles. Resource management, scheduling of jobs, and tracking usage is handled by SLURM.
SLURM supplies user functionality to submit jobs and to check system status. It is also responsible for starting and stopping jobs, collecting job output, and returning output to the user. SLURM allows users to specify attributes about the nodes required to run a given job, for example the CPU architecture.
SLURM is a flexible job scheduler designed to guarantee, on average, that each group or user has the use of the particular number of nodes they are entitled to. If there are competing jobs, processing time is allocated by calculating a priority based mainly on the “fairshare” mechanism of SLURM. On the other hand, if no jobs from other groups are in the queue it is possible for an individual user or group to use a significant portion of the cluster. This maximizes cluster usage while maintaining an equitable sharing. You can find more details about submitting jobs through SLURM in Submitting your first job section or ACCRE’s own SLURM Documentation page.
.
