InfiniBand is an industry-standard communications spec developed by the InfiniBand Trade Association. It defines a switched fabric architecture connecting servers, communications equipment, storage systems, and more within data centers. Let’s take a closer look at what InfiniBand is.
What is InfiniBand? A Closer Look at the Technology
At its most basic, InfiniBand is a type of network that includes servers, storage systems, and other devices. It is popular for applications that require high-performance computing. InfiniBand is known for its low latency and high throughput, making it ideal for sharing data between computers. Some key features include:
- Failover capabilities
- Network layer
- Quality of Service
- Scalability
- Two-layer architecture
Nvidia manufactures InfiniBand host adapters and network switches, which various vendors sell. InfiniBand competes with Ethernet, Fibre Channel, and Intel Omni-Path.
Who Makes InfiniBand?
Nvidia was the largest seller of InfiniBand chips and is the current manufacturer of many InfiniBand components thanks to its 2020 acquisition of Mellanox Technologies, an Israeli-American company that manufactured InfiniBand network switches and adapters. Nvidia has stopped using the Mellanox brand name. Other manufacturers include Cisco, IBM, NADDOD, and QLogic.
How Does InfiniBand Work?
InfiniBand connects devices through a network of switches. This architecture allows for effective communication between servers, storage devices, and compute nodes. It also makes systems highly scalable.
A typical InfiniBand network consists of processor nodes, such as computers, servers, and storage devices, with set host channel adapters. Each device has a target channel adapter to facilitate communication with the other devices.
InfiniBand networks are similar to those supporting many other mainframe computers. All transmissions begin and end with a channel adapter that securely exchanges information with the rest of the system.
Data gets transmitted as messages within data packets in InfiniBand networks, making it ideal for high-performance situations, such as data mining, deep learning for AI, and predictive analytics. InfiniBand supports the following types of communication models:
- Remote direct memory access that bypasses the CPU
- Standard packet-based send-and-receive messages
- Multicast transmissions
- Reversible transaction-based operations
The InfiniBand architecture specs were developed by merging Future I/O and Next Generation I/O. The InfiniBand Trade Association maintains and enhances the specification.
Networkers must connect all IP addresses in a particular subnet to the same InfiniBand subnet. Because they can connect multiple adapters to the same InfiniBand subnet, multiple IP subnets can be on the same network.
InfiniBand is a network specifically designed for remote direct memory access, providing low latency. It does not use TCP for transmission purposes.
What Hardware Does InfiniBand Use?
Key components of InfiniBand architecture include switches and adapters. Host channel adapters enable communication from installation points in servers and network endpoints. Storage devices use target channel adapters for communication. Switches create the network fabric between adapters. Routers, cables, and connectors make up the rest of the hardware.
How is InfiniBand Different From Ethernet?
InfiniBand and Ethernet are both networking technologies, but they have significant differences. InfiniBand is a low latency, high bandwidth technology. It is well-suited for high-performance computing in enterprise applications and data centers that require high data transfer speeds. It can reach speeds of up to 00 Gbps with a latency of under one microsecond.
This technology comes at a premium price compared to Ethernet. However, the upfront investment is often offset by lower upgrade costs. InfiniBand is highly scalable and adaptable, which makes maintaining systems that use it simpler.
Ethernet is generally preferred for general business applications because it is more flexible and cheaper to install. Ethernet-based networks are popular in traditional IT environments. They are cost-effective and compatible with many other technologies. Ethernet also provides reliable data transfer over distances.
While Ethernet technology supports high speeds of 100Gbps or more, it does not offer the exceptionally low latency of InfiniBand. Instead, networks using Ethernet generally experience latency rates of between 10 and 100 microseconds.
How Does InfiniBand Compare to Other Technologies?
InfiniBand has a single-line peak throughput of 800 Gb per second, while CXL reaches peak speeds of 512 Gb per second. InfiniBand latency is about 100ns, while CXL latency is about 500ns. That means that latency is a lot lower in InfiniBand than with CXL.
RoCE v2 is a new technology that can be more flexible than InfiniBand. While both enable the construction of remote direct memory access networks for high-performance computing capabilities, RoCE v2 integrates more seamlessly into traditional Ethernet networks.
NVLink is a point-to-point connection, meaning it connects two devices. InfiniBand is a switched fabric interconnect technology that operates at high speeds and connects multiple devices within a network.
Where Is InfiniBand Commonly Used?
InfiniBand has been implemented extensively in networks that require high-performance computing systems. It is a natural fit for AI systems. Since its creation, it has remained one of the most relevant solutions due to its low latency and no losses.
What is the Bandwidth of InfiniBand?
The initial standards supported data transfer rates of 2.5 GB per second and 10 GB per second. However, the InfiniBand Trade Association raises standards to meet advances in the technology and growing demand for it. As of 2024, InfiniBand networks can reach 800 GB per second.
InfiniBand networks use remote direct memory access for high-speed data transfer between connected devices. These data transfers do not involve the CPU or require intermediate software or hardware. That means that data does not need to be copied, which provides higher speeds.
InfiniBand uses link-level flow control to ensure transfers do not drop packets in the network fabric. This flow control makes it lossless and contributes to efficient bandwidth usage within data centers. This technology makes InfiniBand a preferred choice for long-range communication between data centers.
What Will You Do Now That You Know the Answer to What Is InfiniBand?
Equal Optics provides top-tier networking solutions for data centers and enterprise-level networks. We have all of the components you need to create a network using InfiniBand technology. Answering “What is InfiniBand?” is the first step in making an informed decision about whether it is the right technology for your network. Contact us today to get started with your setup.
