is easy. You load the weights, load the information, and anticipate it to complete. For many fashions, that’s nice, and the one actual value is time. When the wait will get too lengthy, the same old repair is so as to add GPUs. Each trains on a special slice of the information in parallel, and the work finishes sooner. Nothing in regards to the mannequin state adjustments; you’ve simply thrown extra palms at it.
That covers the frequent case, the place the issue is time. Massive fashions add a second downside that extra palms can’t clear up: house. Over the previous yr, a serious a part of my work has concerned coaching and fine-tuning massive basis fashions throughout totally different domains, together with single-cell fashions and vision-and-language transformer fashions. When you get previous just a few billion parameters, the mannequin, its gradients, and its optimizer states not match on a single GPU. You’ll be able to’t simply replicate the work throughout GPUs as a result of a duplicate doesn’t match on any of them. It’s important to break up the mannequin itself.
These two issues map onto two methods. DDP (Distributed Information Parallel) assaults time. Each GPU retains a full copy of the mannequin and trains on a special slice of the batch. FSDP (Absolutely Sharded Information Parallel) assaults house. The mannequin is break up into items in order that no GPU has to carry the whole mannequin.
There aren’t actually simply two choices, although. These are simply the 2 ends of 1 dial. In between these sit the ZeRO levels from DeepSpeed (Microsoft’s coaching library), which allow you to slide from one finish to the opposite, step-by-step, buying and selling a bit of reminiscence for a bit of communication at every cease moderately than leaping straight from “all people holds every thing” to “no person does.”
The place to set that dial and what it really controls is roofed within the first a part of the article. However there’s a second factor that decides how properly any of those methods work. Every one in every of them works by shifting knowledge between GPUs. And how briskly that knowledge strikes will depend on the bodily wires connecting them. This was one thing I didn’t take into consideration till the identical code, on the identical type of GPUs, ran a number of occasions slower on one node than on one other. So within the second half, we’ll talk about the {hardware} beneath it, and the way the wiring must also be one thing you think about. (For people who are usually not a lot into {hardware}, belief me, it’s not that unhealthy!)
One word earlier than we begin. Every little thing right here is measured, not simply concept. The technique comparisons ran on 4 A100 80GB GPUs; the {hardware} experiments ran on H200 machines, together with two which have the very same GPUs wired collectively in numerous methods, which seems to be the entire level.
Half 1: The Software program: What Every GPU Holds
Each distributed coaching technique is de facto a solution to at least one query: what does every GPU preserve its personal copy of? For that to make sense, let’s have a look at an instance.
Take Mistral-7B and fine-tune it with Adam in BF16 combined precision. The parameters take 14 GB: 7 billion of them, two bytes every. The gradients take one other 14 GB. Then now we have the optimizer states, that are the costliest half. Adam retains two operating averages for each parameter, momentum and variance, each saved in FP32, which works out to roughly 58 GB. Add it up, and we attain 87 GB earlier than the mannequin has seen a single batch or a single activation has been computed.
For an A100 GPU that has 80 GB of VRAM, the mannequin doesn’t match. So we distribute. And the query, as talked about at the start of this part, is: what does every GPU maintain?
Distributed Information Parallel (DDP): Everybody Holds Every little thing
The easy reply is that each GPU holds all of it. That is DDP, and it’s the one most individuals first encounter (which was the case for me as properly). Every GPU retains a whole copy of the mannequin: all of the parameters, all of the gradients, all of the optimizer states. What will get divided is the information. Every GPU takes a special slice of the batch, runs its personal ahead and backward cross, and computes gradients from its personal slice.
The catch is that these gradients are all totally different as a result of every GPU noticed totally different knowledge. If each GPU simply stepped its optimizer now, the copies would drift aside, and also you’d not have one mannequin, you’d have 4. So earlier than stepping, the GPUs common their gradients. Each GPU sends its gradients out, receives everybody else’s, and so they all find yourself with the identical averaged gradient. They then take an equivalent optimizer step and the copies keep in sync. That averaging is the one piece of communication DDP wants, and it occurs as soon as per step. It’s referred to as all-reduce, and we’ll come again to it later after we begin counting what every technique prices the wire.
DDP is quick and easy exactly as a result of it communicates so little. One all-reduce per step, and PyTorch overlaps even that with the backward cross, so on good {hardware}, you barely discover it. However its limitation is constructed into the design. Each GPU holds every thing, which suggests including GPUs does nothing for the reminiscence downside. If 87 GB doesn’t match on one A100, it doesn’t match on 4 both, as a result of every of the 4 remains to be holding the complete 87 GB. Including GPUs to DDP buys you throughput, however it doesn’t scale back the reminiscence every GPU has to carry.
Absolutely Sharded Information Parallel (FSDP): No one Holds Every little thing
If the issue is that each GPU holds every thing, the apparent repair is to ensure no GPU does. That is FSDP. As an alternative of every GPU preserving a full copy, the mannequin is break up into items, and every GPU owns only one piece of the parameters, one piece of the gradients, and one piece of the optimizer states. On 4 GPUs, every one holds roughly 1 / 4 of the 87 GB, which permits the mannequin to suit.
However this raises a right away downside. You’ll be able to’t run a layer by means of a GPU that solely has 1 / 4 of that layer’s weights. So FSDP reassembles every layer simply earlier than it’s wanted and takes it aside once more proper after. When the ahead cross reaches a layer, the GPUs commerce their items so that each GPU briefly has that layer’s full weights, the layer runs, after which every GPU throws away the items it doesn’t personal. The following layer comes up and the identical factor occurs once more. The complete mannequin is rarely resident on any single GPU; just one layer is, and solely for the immediate it’s computing. The identical factor occurs in reverse through the backward cross.
That buying and selling has names, the identical method DDP’s averaging did. Gathering everybody’s items right into a full layer is an all-gather, and scattering the summed gradients again to their house owners is a reduce-scatter. Take into account this distinction. DDP communicates as soon as per step. FSDP communicates always. An all-gather earlier than each layer within the ahead cross, one other earlier than each layer within the backward cross, and a reduce-scatter after. That’s mainly the commerce that you simply do when switching between them. FSDP buys reminiscence and pays for it in communication.
One clarification issues right here, as a result of it may very well be type of complicated. FSDP remains to be knowledge parallelism. Each GPU runs the whole mannequin by itself slice of the batch, precisely like DDP. The one distinction is that the weights are saved in items and reassembled on demand as an alternative of being stored as a full copy. Every GPU nonetheless computes the entire ahead and backward cross from begin to end.
That is totally different from mannequin parallelism, the place the mannequin itself is break up throughout GPUs, and every one is chargeable for solely a part of the computation, one set of layers, or one slice of each layer. On this paradigm, a single ahead cross has to hop between GPUs as a result of no single one can end it alone. So as to add to this, even inside mannequin parallelism, now we have layer and tensor parallelism, which dictate whether or not we divide the mannequin vertically or horizontally among the many layers. However earlier than we digress an excessive amount of, these are totally different strategies with totally different communication patterns, and it’s not what this text is about. Every little thing right here, DDP, FSDP, and the ZeRO levels in between, is knowledge parallel. Similar computation on each GPU, simply with totally different knowledge. Solely the storage of the mannequin differs from one technique to the subsequent.
Coming again to it, now that we’ve mentioned the speculation, let’s see what occurs in observe. To see how these two methods examine, I ran experiments on two fashions. DINOv2 (a imaginative and prescient basis mannequin) and Mistral-7B (a big language mannequin). Each experiments ran on 4 A100 80GB GPUs on a single node, utilizing HuggingFace Speed up with BF16 combined precision. The one factor that modified between runs was the technique.
DINOv2 on Meals-101. The primary experiment fine-tunes DINOv2 on Meals-101, a classification dataset with 101 meals classes and about 101k photos. The duty is to foretell a picture class given a picture. I examined each the bottom (86M parameters) and enormous (304M parameters) backbones, utilizing a batch dimension of 32 per GPU for the bottom mannequin and 256 per GPU for the big mannequin. This can be a case the place each methods work because the mannequin matches comfortably in GPU reminiscence. However even right here, we are able to see the reminiscence distinction between the 2 approaches.
FSDP makes use of considerably much less reminiscence per GPU: 0.39 GB vs. 1.79 GB for the bottom mannequin, and 1.39 GB vs. 6.26 GB for the big mannequin. That’s roughly 4.5x much less reminiscence with FSDP. This issues much less when you might have loads of headroom, however it exhibits us why FSDP can be important for bigger fashions. So we noticed the reminiscence, what in regards to the velocity?
We are able to see that DDP is persistently sooner, round 6% larger throughput for the bottom mannequin and 4% for the big one. That is anticipated since DDP has much less communication overhead. It simply has one all-reduce per step, in opposition to FSDP’s gather-and-discard at each layer. And after we see the coaching time:
The takeaway right here is that when the mannequin matches in reminiscence, DDP is the higher alternative. It’s less complicated and sooner. However discover the reminiscence hole. What occurs after we scale as much as a mannequin the place that hole really issues?
Mistral-7B on Alpaca. For the second half, I attempted fine-tuning the Mistral mannequin on the Alpaca instruction-following dataset. With DDP, the job crashed instantly with an out-of-memory error. With FSDP, it ran with out a problem.
DDP would wish an estimated 87 GB per GPU for the mannequin state (parameters, gradients, optimizers), which exceeds the 80 GB accessible on the A100. FSDP shards this throughout 4 GPUs, bringing the estimated requirement all the way down to about 22 GB per GPU. The precise peak through the run was about 29 GB, which incorporates activation and different overhead. Because of this FSDP exists. It makes coaching doable when DDP merely can’t match the mannequin.
In order that’s the rule I used to cease at. Does the mannequin, with its gradients and optimizer states, match on one GPU? If sure, use DDP. If no, use FSDP. It’s a very good rule. However there are two assumptions hiding inside it. The primary is that DDP and FSDP are the one two settings. To start with, I already hinted that they aren’t. There’s a dial between them, and within the subsequent half, we’ll talk about these levels, the place you shard a number of the mannequin state however not all of it.
The second assumption is that we’re treating all machines the identical method. These methods don’t behave the identical all over the place. The identical technique on the identical mannequin can run a number of occasions sooner or slower relying on how the GPUs are wired collectively. That’s what the second half of the article is about. First, although, the dial.
ZeRO: The Dial Between DDP and FSDP
To see the stops on the dial, begin with a small truth in regards to the operations from Half 1. DDP’s all-reduce, the one which averages gradients throughout GPUs, is definitely two less complicated operations glued collectively. First is reduce-scatter, which sums everybody’s gradients however offers every GPU solely its slice of the outcome as an alternative of giving everybody the complete outcome. Then an all-gather, which collects these slices again right into a full copy on each GPU. So all-reduce is mainly reduce-scatter plus all-gather.
These are the identical two operations that FSDP makes use of. Now that we all know that the operations are shared, it is sensible that the quantity of sharding may very well be in levels. The dial is then only a query of how a lot you’re keen to maintain sharded between steps as an alternative of reassembling.
To make that concrete, have a look at what every GPU shops per parameter. In mixed-precision Adam, there are three issues: the parameters, the gradients, and the optimizer states, with the optimizer states being by far the heaviest. DDP retains full copies of all three on each GPU. The ZeRO levels, from DeepSpeed, peel them off one after the other, heaviest first.
ZeRO-1 shards solely the optimizer states. Every GPU nonetheless holds the complete parameters and full gradients, however it retains solely its slice of the optimizer states. Since these are the heaviest gadgets, this provides the most important reminiscence win, and it prices little or no in additional communication, as a result of nothing you want through the ahead and backward cross has moved.
ZeRO-2 provides the gradients to the shards. Now every GPU retains solely its slice of the gradients and its slice of the optimizer states, however nonetheless a full copy of the parameters. Communication remains to be near DDP’s, because the parameters (the factor you really compute with) are all nonetheless native.
ZeRO-3 provides the parameters as properly. Now, no GPU holds the complete mannequin, so earlier than a layer can run, its parameters should be gathered on the fly, used, and discarded. ZeRO-3 and FSDP implement the identical primary concept. One is constructed by DeepSpeed, and the opposite is native to PyTorch. So, since that is the place you cease preserving the parameters native, additionally, you will have the additional communication we mentioned earlier.
The next diagram exhibits the dial from full replication to full sharding:
Two issues within the determine are value pausing on. The primary is the reminiscence column on the precise. These are the measured peaks for Mistral-7B on 4 GPUs, and so they fall in a clear sample as you progress down the dial. 87 GB at DDP, then 55, 51, 40, and 37 GB. However discover it by no means drops to 1 / 4 of 87 GB, which is what you’d count on if 4 GPUs every held 1 / 4 of every thing. The ground is usually activations and runtime overhead. Activations are nonetheless produced independently on every GPU, so sharding the mannequin state doesn’t make them disappear.
Second, each step down that dial buys reminiscence by spending communication. ZeRO-1 spends nearly none. ZeRO-3 and FSDP spend probably the most. That’s the software program view. DDP, ZeRO, and FSDP are totally different solutions to the identical query: what ought to every GPU preserve domestically, and what ought to it fetch from the others? Transferring towards sharding buys reminiscence, however it additionally creates extra communication.
To date, now we have solely counted how a lot knowledge has to maneuver. We’ve not requested how costly that motion is. That will depend on the {hardware}. Extra particularly, it will depend on the material connecting the GPUs.
Half 2: The {Hardware}: How Quick the GPUs Can Discuss
The Wires Between the GPUs
Each technique in Half 1 had the identical hidden value line: communication. DDP pays it as soon as per step. FSDP pays it at each layer. ZeRO permits you to select how a lot to pay. However communication is just not an summary value. It’s knowledge shifting between chips, and the trail it takes issues.
On this article, I’m solely communication inside a single node. As soon as coaching spans a number of servers, one other layer seems. GPUs now have to speak throughout InfiniBand or Ethernet, which brings a special set of bottlenecks. That issues rather a lot as properly, however it’s a separate article.
Inside one node, when GPUs have to trade knowledge, how do they do it? Nicely, there are totally different roads they will take, and the velocity between them can differ by as much as 10×. The identical FSDP code can run at full velocity on one machine and crawl on one other with the very same GPUs, purely due to how these GPUs are related to one another.
There are two primary paths GPU site visitors can take inside a server: PCIe and NVLink. PCIe is the default system path by means of the motherboard. NVLink is NVIDIA’s direct GPU-to-GPU interconnect.
However NVLink is just the hyperlink. The topology issues simply as a lot. On the machines I examined, NVLink appeared in two totally different preparations. NVL bridge teams and NVSwitch. That provides us 4 phrases to maintain straight. PCIe, NVLink, NVL, and NVSwitch. They’re associated, however they don’t seem to be the identical type of factor. PCIe and NVLink are the paths. NVL and NVSwitch are methods of arranging the quick NVLink path throughout a number of GPUs.
PCIe: the default connection
PCIe (PCI Categorical) is the usual high-speed bus that connects parts to a pc’s motherboard, the slots your graphics card, SSD, and community all plug into. Each GPU sits on it by default.
When two GPUs have to trade knowledge over PCIe, it goes by means of the system: out of 1 GPU, throughout the shared bus, usually by means of the CPU and host reminiscence, and into the opposite. It really works between any two units, however it’s the slowest possibility, and each machine on the bus is contending for a similar lanes. A PCIe Gen5 x16 slot strikes someplace round 64 GB/s in a single route.
NVLink: a direct GPU-to-GPU connection
NVLink is NVIDIA’s devoted interconnect. Hyperlinks that run from one GPU to a different with out going by means of the CPU or system reminiscence. On an H100 or H200, every GPU has 18 NVLink connections, which collectively come to roughly 450 GB/s in every route, about 7x what a single PCIe slot gives.
However NVLink is just the hyperlink. What issues simply as a lot is how these hyperlinks are organized throughout the GPUs in a server. Two machines can each have H200 GPUs and nonetheless behave very in a different way if one makes use of NVLink by means of bridges and the opposite makes use of NVSwitch.
NVL: hyperlinks inside teams, PCIe between them
The cheaper association connects NVLink solely inside small teams of GPUs, utilizing bodily bridges between playing cards. NVIDIA sells these as “NVL” playing cards, just like the H200 NVL. On the machines I examined, eight playing cards had been bridged into two teams of 4. Inside a bunch, each pair of GPUs has a quick NVLink bridge. Between teams, there’s no NVLink in any respect, and site visitors falls again to PCIe.
So the material is uneven. Some pairs of GPUs speak over NVLink, others speak over PCIe. And so which pair you get will depend on which bodily GPUs your job lands on, which is set by the cluster scheduler. On an NVL machine, placement is necessary as it may well have an effect on efficiency.
NVSwitch: each GPU related to each different
The premium association places a swap within the center. Devoted NVSwitch chips join each GPU to each different GPU at full NVLink velocity, with no quick pairs and no gradual pairs. Each GPU is one hop away from each different. That is what’s on NVIDIA’s HGX baseboards, the SXM type issue, and it’s the structure that enormous coaching runs use.
To get a way of how far this concept scales, at Computex 2025, Jensen Huang confirmed off what NVIDIA calls an NVLink Backbone, a column of round 5000 cables wiring 72 GPUs right into a single all-to-all cloth, and claimed it strikes about 130 TB/s. That is greater than the height site visitors of the whole web. Take the comparability as you’ll, however the underlying level is actual. An NVSwitch cloth is a crossbar the place each GPU reaches each different at full velocity, and it scales to an entire rack. The machine I examined is a a lot smaller model of the identical concept. It has 8 GPUs as an alternative of the 72, however wired on the identical precept.
Similar GPU, two totally different machines
It’s value clearing up a confusion that may occur. After we say “H200,” we’re principally naming the GPU. H200 techniques could be constructed in numerous type elements and materials: as SXM GPUs on an NVSwitch baseboard, or as NVL playing cards related by bridges. Each are H200-class GPUs with 141 GB of reminiscence, however they don’t seem to be the identical system. Energy limits, board design, and cooling can differ. However the level right here is that you simply can not infer the GPU-to-GPU cloth from the GPU mannequin title alone. It’s important to have a look at the topology. Listed here are the 2 machines used for the experiments beneath. You’ll be able to verify the wiring in your node with nvidia-smi topo -m.
On the SXM node, each pair reads NV18. That is uniform, full velocity, each GPU is equal. On the NVL node, the identical H200 playing cards break up into two teams of 4, quick NVLink inside a bunch and a plain PCIe throughout. A job that lands on 4 GPUs inside one group runs close to full velocity. A job scattered throughout teams drops all the way down to PCIe. The following part is about precisely what that prices.
Measuring the Wires
We’ve two sorts of measurements right here. The primary is a microbenchmark that occasions the precise collective operations the methods rely on. All-reduce for DDP, all-gather and reduce-scatter for FSDP, and reviews the bandwidth every one achieves. The second is end-to-end coaching, the identical Mistral-7B fine-tune from Half 1, now timed for steady-state throughput, so we are able to see whether or not the bandwidth variations really present up in coaching. Every little thing beneath ran on the 2 machines described above.
Let’s begin with the cleanest doable comparability. Take two H200 SXM GPUs and run the identical collectives twice. For the primary run, allow them to use NVLink, and for the second, drive them onto PCIe as an alternative. Similar GPUs, similar code, solely the wire adjustments.
The wire alone is value about 10-11x. This is identical silicon and the identical operation, the place the one distinction was the bodily connection the information travels over.
Subsequent, I measured how all-reduce bandwidth adjustments as you add GPUs to the job. For this, I needed to management which bodily GPUs every run used, so I may preserve a job inside a single NVL group or drive it to cross between teams.
We are able to see three behaviors right here. NVSwitch stays flat and quick. 330 GB/s at two GPUs, rising gently to 467 at eight, and it by no means issues which bodily GPUs the job will get. Each pair is NV18, so the material seems the identical regardless of how the scheduler fills it. That is the conduct massive coaching runs assume.
The NVL quad is a shock, and it comes all the way down to how NVLink hyperlinks are shared. Every of those GPUs has 18 NVLink hyperlinks to distribute amongst its neighbors. Inside a quad of 4, each GPU has three neighbors and offers six hyperlinks to every one, which is the NV6 you’d see within the topology matrix. Six hyperlinks per pair, eighteen hyperlinks complete per GPU.
That sharing is why job dimension issues a lot. A job utilizing solely two GPUs of the quad talks throughout simply the six hyperlinks connecting that one pair. The opposite 12 hyperlinks on every card go to GPUs that aren’t within the job, in order that they sit idle, and the pair runs at about 114 GB/s, which is a 3rd of the cardboard’s NVLink capability. Add the opposite two GPUs, and each card now talks to all three of its neighbors without delay, lighting up all 18 hyperlinks. That’s the identical variety of hyperlinks an SXM module makes use of, which is why a full quad reaches about 322 GB/s, practically matching NVSwitch.
So the NVL quad was by no means gradual. It was simply underused. A small job leaves most of its hyperlinks idle. After we fill the quad, we fill the hyperlinks.
The third line is the cross quad case, and it by no means leaves PCIe territory. 28 GB/s at two GPUs, 35 at 4, 39 at eight. Including GPUs doesn’t assist as a result of the issue isn’t what number of hyperlinks are energetic; it’s that the job has to cross between quads, and there’s no NVLink there in any respect. A collective is just as quick as its slowest required hop, so a single PCIe hyperlink between the quads drags the whole all-reduce all the way down to PCIe velocity, regardless of how briskly each different hyperlink runs. And since a quad is just 4 GPUs broad, any job bigger than 4 on this machine is compelled to cross. Eight GPUs can’t keep away from it.
The clearest view is the 4-GPU level on the determine. The identical 4 playing cards, in the identical node, run at 322 GB/s inside one quad and about 35 GB/s throughout two, a roughly 9× distinction determined by nothing however which GPUs the job landed on. On an NVL machine, placement actually does make a distinction.
What it does to actual coaching
The bandwidth numbers show the wire issues in a microbenchmark. I additionally needed to see if it exhibits up in actual coaching. So I ran the identical Mistral-7B fine-tune from Half 1 on 4 GPUs, 3 ways: on NVSwitch, on a full NVL quad, and on an NVL job intentionally spanning two quads.
Take a look at the 2 quick materials first, NVSwitch and the complete NVL quad. They’re practically equivalent for each technique. The playing cards had been by no means the bottleneck, it was the location. (If FSDP seems prefer it edges out DDP contained in the quad, don’t learn an excessive amount of into it. Repeat runs put the noise at just a few p.c, so these are a tie. The sincere abstract is that, on both quick cloth, DDP and FSDP are even.)
The across-quads case is the price of getting placement mistaken. Throughput drops by 3 to 5x. That’s a smaller hit than ~10x bandwidth drop from determine 9, and the hole is value understanding. Coaching isn’t pure communication. Every GPU remains to be doing actual compute that doesn’t care in regards to the wire, and that compute partly hides the gradual communication behind it. So the wire’s full penalty will get considerably diluted. However 3 to 5x remains to be a really significant distinction, particularly for lengthy jobs.
And the slowdown isn’t even throughout the methods. The sample is strictly what Half 1 predicts. FSDP falls the toughest, as a result of it communicates each layer, and on a gradual wire, these per-layer transfers can’t conceal behind compute. DDP holds up finest as a result of it communicates simply as soon as per step. When the wire is quick, sharding is almost free, so FSDP retains tempo with DDP. When the wire is gradual, each additional little bit of communication is uncovered, so the extra a method shards, the extra it loses.
Put all of it collectively. The entire dial, on three wires
Earlier, we mentioned that DDP and FSDP are two ends of 1 dial, with ZeRO-1, -2, and -3 because the stops between them. That was a diagram, and now I can measure it. Right here’s the identical Mistral-7B job on 4 GPUs, run at each cease on the dial, on a quick wire and on gradual ones.
Begin with the reminiscence panel on the precise as a result of it does precisely what the speculation promised. Transferring down the dial, peak reminiscence falls in a clear sample. 87 GB at DDP, then 55 at ZeRO-1, 51 at ZeRO-2, and 40 at ZeRO-3. Every cease shards another factor, and every one buys headroom. FSDP sits on the backside at 37 GB, however discover it’s not an enormous step down from ZeRO-3’s 40, as a result of FSDP and ZeRO-3 shard precisely the identical issues. As I discussed above, they shard the identical issues, however the frameworks implement and schedule issues in a different way, which explains the small hole that we see. This staircase is the half that at all times works, no matter {hardware}. In case your downside is becoming the mannequin, the dial delivers, predictably.
The throughput panel is attention-grabbing as a result of the dial behaves like two various things relying on the wire.
On the quick wire, the center of the dial loses. ZeRO-1 and ZeRO-2 quit round 20% of DDP’s throughput, and ZeRO-3 offers up extra. However FSDP, sharding the identical issues as ZeRO-3, claws practically all of that throughput again whereas utilizing about the identical reminiscence. That’s attention-grabbing as a result of the final two are implementations of the equivalent algorithm, and land far aside on a quick wire.
The distinction is implementation, not technique. The 2 frameworks schedule and overlap their communication in a different way, and on a quick wire, the place communication is reasonable, that overhead is the primary factor you’re measuring. (That is Hugging Face speed up’s default DeepSpeed config, with no tuning. A tuned setup would possible slim the hole; the purpose is what the defaults provide you with, not that one framework is inherently slower.)
On a gradual wire, the dial flattens. Every little thing collapses towards the identical low throughput, and the variations between levels principally wash out. The costly factor on a gradual wire is communication itself, and as soon as that dominates, the query of which precise tensors you shard barely strikes the quantity. On this case, the wire is setting the tempo, not the technique.
Placing every thing above collectively, we are able to now make a extra knowledgeable resolution.
When you’re on NVSwitch (SXM), cease serious about the wire. Each GPU is equally quick, and any GPUs the scheduler offers you’re nearly as good as another. Use DDP if the mannequin matches, FSDP if it doesn’t. And since FSDP matches DDP’s velocity right here whereas utilizing far much less reminiscence, it’s an inexpensive default even when the mannequin does match.
When you’re on NVL and your job matches inside one bridged group, pin it there, and also you get near-NVSwitch velocity. On the machines I examined, which means 4 GPUs, and also you management the location with CUDA_VISIBLE_DEVICES. FSDP is a powerful default on this case. It will get you nearly DDP-class velocity at roughly half the reminiscence.
When you’re on NVL and your job has to span teams, count on a substantial slowdown, worst for the communication-heavy methods. Don’t count on a milder ZeRO stage to win the velocity again both, as a result of the dial flattens on a gradual wire. If the mannequin matches replicated, plain DDP is the least-bad possibility. If it doesn’t, choose the stage that matches in reminiscence and simply settle for that you simply’re going to pay the wire’s worth.
Conclusion
We’ve lined numerous floor. Within the first half, we regarded on the totally different methods. DDP, ZeRO, and FSDP. Every of them solutions the identical query another way: what ought to every GPU retailer domestically, and what ought to it fetch from the others? DDP retains every thing replicated. FSDP shards every thing. ZeRO offers you the stops in between.
Within the second half, we regarded on the {hardware} layer, the material these requests journey over. On a quick cloth, sharding could be low-cost sufficient that FSDP retains tempo with DDP whereas utilizing a lot much less reminiscence. On a gradual cloth, the identical additional communication turns into seen, and may gradual issues down considerably.
That’s the primary lesson. The technique decides how a lot knowledge strikes, however the wire decides how costly that motion is. So earlier than launching a distributed job, run one command:
nvidia-smi topo -m
It tells you whether or not your GPUs are related by NVLink, NVSwitch, or a slower host path.
Thanks for studying, and I hope you discovered this useful!
References and Additional Studying
