AI Card Bao KDOKFMPPVPDEVELOPER.AICARDBAO.COM

Industry News

Three mini hosts are connected with USB4 cables to run the 235B model. Microsoft releases a preview of RDMA over USB4

Microsoft has released a developer preview of RDMA over USB4 for Windows.

In simpler terms: when several computers are connected using ordinary USB4 cables, they can share their memory and GPU power to jointly process the same large-scale local model. No dedicated network cards or switches are required for this process.

For the demonstration, three AMD Ryzen AI Max+ 395 processors were used to run the 4-bit quantized Qwen3-235B-A22B model, achieving a speed of approximately 18.5 tokens per second.

The number 18.5 is not particularly fast, but it is sufficient. The average reading speed of a normal person is also only a few words per second. At this speed, there is no problem engaging in conversations; however, performing batch tasks will be somewhat slower.

The key lies in what it processes. For a model with the 235B parameter setting, after 4-bit quantization, the data related to light weights alone requires over 100 GB of memory—something that a typical consumer-grade computer simply cannot accommodate. However, by combining three mini-machines each equipped with sufficient memory capacity, it is just enough to handle the requirement.

RDMA is a technology that has been around for a long time in data centers. It allows one machine to directly read from and write to the memory of another machine, bypassing the network protocol stack of the operating system, thus resulting in extremely low latency. In the past, to use this technology, one would need to rely on InfiniBand or specialized Ethernet cards – such solutions often cost more than the computer itself. Nowadays, with USB4, the cables required for implementation only cost a few dozen yuan.

Apple has already implemented similar approaches for some time – using Thunderbolt cables to connect multiple Mac computers together in a cluster to run models. This approach has become quite popular within the community of people who use local computing setups. The Windows camp, on the other hand, has always lacked such functionality and has had to rely on regular networks, which results in significant speed losses.

This feature will be released through the Windows Insider program and is currently available only to developers.

For those who want to run large-scale models locally, this provides an additional option for upgrading their systems. In the past, when insufficient memory was an issue, the only solution was to replace the entire machine; now, it’s possible to simply add another identical device to the existing setup. Given that both graphics cards and memory are becoming more expensive, assembling a small cluster consisting of two or three mid-range machines—each capable of running large models with over 200 billion parameters—is far more practical than trying to build a top-of-the-line workstation alone.

What it addresses is the issue of whether enough space is available for installation; however, the speed remains limited by the computing power of a single machine.

Related