CNN-based mannequin extra light-weight? Simply take the smaller model of that mannequin, proper? Like with ResNet, as an illustration, if ResNet-152 feels too heavy, why not simply use ResNet-101? Or within the case of DenseNet, why not go together with DenseNet-121 fairly than DenseNet-169? — Sure, that’s true, however you would need to sacrifice some accuracy for that. Principally, if you would like a lighter mannequin then you need to anticipate your accuracy to drop as properly.
Now, what if I informed you a few mannequin that’s extra light-weight than its base however can nonetheless compete on accuracy? Meet CSPNet (Cross Stage Partial Community). You’ll be shocked that it could successfully scale back computational complexity whereas sustaining excessive accuracy — no tradeoff! On this article we’re going to speak concerning the CSPNet structure, together with the way it works and the way to implement it from scratch.
A Temporary Historical past of CSPNet
CSPNet was first launched in a paper titled “CSPNet: A New Spine That Can Improve Studying Functionality of CNN” written by Wang et al. again in November 2019 [1]. CSPNet was initially proposed to deal with the constraints of DenseNet. Regardless of already being computationally cheaper than ResNet, the authors thought that the computation of DenseNet itself continues to be thought of costly. Check out the principle constructing block of a DenseNet in Determine 1 beneath to grasp why.
In a DenseNet constructing block — known as dense block — each convolution layer takes info from all earlier layers, inflicting it to have a number of redundant gradient info that makes coaching inefficient. We are able to consider it like a scholar taught by 5 totally different lecturers for a similar materials. It’s truly good for the reason that scholar can get a number of views about that particular matter. Nevertheless, in some unspecified time in the future it turns into redundant and thus inefficient. Within the case of DenseNet, we are able to see the deeper layers as college students and all of the tensors from shallower layers as lecturers. Within the instance above, if we assume H₄ as our scholar, then the x₀, x₁, x₂, and x₃ tensors act because the lecturers. Right here you possibly can simply think about how that scholar would get overwhelmed by all that info!
Earlier than we get into CSPNet, I even have a complete separate article particularly speaking about DenseNet (reference [3]), which I extremely advocate you learn if you would like the total image of how this structure works.
Aims
The target of CSPNet is to allow a community to have cheaper computational complexity and higher gradient mixture. The explanation for the latter is that almost all gradient info in DenseNet consists of duplicates of one another. You will need to notice that CSPNet shouldn’t be a standalone community. As a substitute, it’s a new paradigm we apply to DenseNet.
Now let’s check out Determine 2 beneath to see how CSPNet achieves its targets. You may see the illustration on the left that the variety of characteristic maps step by step will increase as we get deeper into the community. In case you have learn my earlier article about DenseNet, that is basically one thing we are able to management by way of the progress price parameter, i.e., the variety of characteristic maps produced by every convolution layer inside a dense block. The truth is, this improve within the variety of characteristic maps is what the authors see as a computational bottleneck.
By making use of the Cross Stage Partial mechanism, we are able to mainly make the computation of a DenseNet cheaper. If we check out the illustration on the proper, we are able to see that we now have a further department popping out from x₀ that goes on to the so-called Partial Transition Layer. There are not less than two benefits we get with this mechanism, that are in accordance with the targets I discussed earlier. First, we are able to save plenty of computations for the reason that variety of characteristic maps processed by the dense block is barely half of the unique one. And second, the gradient info turns into extra various since we obtained a further path with unprocessed characteristic maps that avoids the redundant gradient info. So briefly, the thought of CSPNet eliminates the computational redundancy of DenseNet (by way of the skip-path) whereas on the similar time nonetheless preserves its feature-reuse property (by way of the dense block).
The Detailed CSPNet Structure
Talking of the main points, the unique characteristic map is first divided into two elements in channel-wise method, the place every of them will probably be processed in numerous paths. Suppose we obtained 64 enter channels, the primary 32 characteristic maps (half 1) will skip by way of all computations, whereas the remaining 32 (half 2) will probably be processed by a dense block. Though this splitting step is fairly simple, the merging step is definitely not fairly trivial. You may see in Determine 3 beneath that we obtained a number of totally different mechanisms to take action.
Within the construction known as fusion first (c), we concatenate the half 1 tensor with the half 2 tensor that has been processed by the dense block previous to passing them by way of the transition layer. So, possibility (c) is definitely fairly simple to implement as a result of the spatial dimension of the 2 tensors is precisely the identical, permitting us to concatenate them simply.
In my earlier article [3], I discussed that the transition layer of a DenseNet is used to cut back each the spatial dimension and the variety of channels. The truth is, this property requires us to rethink the way to implement the fusion final (d) construction. That is basically as a result of the transition layer will trigger the half 2 tensor to have a smaller spatial dimension than the half 1 tensor. So technically talking, we have to both apply one thing like a pooling with a stride of two to the half 1 department or just omitting the downsampling operation within the transition layer. By doing this, the spatial dimension of the 2 tensors would be the similar, and thus they’re now concatenable.
As a substitute of simply utilizing a single transition layer positioned both earlier than or after characteristic mixture, the authors additionally proposed one other methodology which they seek advice from as CSPDenseNet (b). We are able to consider this as a mixture of (c) and (d), the place we obtained two transition layers positioned earlier than and after the tensor concatenation course of. On this explicit case, the primary transition layer (the one positioned within the half 2 department) will carry out channel discount by cross-channel pooling, i.e., a pooling layer that operates throughout channel dimension. In the meantime, the second transition layer will carry out each spatial downsampling and channel depend discount. So mainly, on this strategy we scale back the variety of channels twice — properly, not less than that’s what I perceive from the paper concerning the two transition layers, because the detailed processes inside these layers aren’t explicitly mentioned.
Experimental Outcomes
Speaking concerning the experimental outcomes relating to these characteristic mixture mechanisms, it’s defined within the paper that fusion final (d) is best than fusion first (c), the place the previous can considerably scale back computational complexity whereas solely suffers from a really slight drop in accuracy. Variant (c) truly additionally reduces computational complexity, but the degradation in accuracy can also be vital. Authors discovered that variant (b) obtained an excellent higher outcome than the 2. Determine 4 beneath shows a number of experimental outcomes exhibiting how the three characteristic mixture mechanisms carried out in comparison with the bottom mannequin. Nevertheless, as a substitute of utilizing DenseNet, they in some way determined to make use of PeleeNet to check these constructions.
Based mostly on the above determine, we are able to see that the CSP fusion final (inexperienced) certainly performs higher in comparison with the CSP fusion first (purple). That is primarily based on the truth that its accuracy solely degrades by 0.1% from its base mannequin whereas having 21% smaller computational complexity. In the meantime, regardless that CSP fusion first efficiently reduces computational complexity by 26%, however the accuracy drop is fairly vital because it performs 1.5% worse than the bottom PeleeNet. And probably the most spectacular construction is the CSPPeleeNet variant (blue), i.e., the one which makes use of two transition layers. Right here we are able to clearly see that though the computational complexity is lowered by 13%, the accuracy of the mannequin truly improves by 0.2% — once more, no tradeoff!
Not solely that, however the authors additionally tried to implement CSPNet on different spine fashions. The ends in Determine 5 beneath reveals that the CSPNet construction efficiently reduces the computational complexity of DenseNet -201-Elastic and ResNeXt-50 by 19% and 22%, respectively. It’s fascinating to see that the accuracy of the ResNeXt mannequin improves regardless of the discount in mannequin complexity, which is in accordance with the outcome obtained by CSPPeleeNet in Determine 4.
The Mathematical Expression of CSPDenseNet
For individuals who love math, right here I obtained you some notations that you just may discover fascinating to know. Figures 6 and seven beneath show the mathematical expressions of DenseNet and CSPDenseNet blocks through the ahead propagation section.
Within the DenseNet block, x₁ corresponds to the tensor produced by the primary conv layer w₁ primarily based on the enter tensor x₀. Subsequent, we concatenate the unique tensor x₀ with x₁ and use them because the enter for the w₂ layer (or to be extra exact, w is definitely the weights of the conv layer, not the conv layer itself). We hold producing extra characteristic maps and concatenating the present ones as we get deeper into the community. On this approach, we are able to mainly say that the outputs of all earlier layers turn out to be the enter of the present layer.
The case is totally different for CSPDenseNet. You may see within the notation beneath that we obtained x₀’ and x₀’’, which we beforehand seek advice from because the half 1 and half 2. The x₀’’ tensor undergoes processing just like the one in DenseNet block till we obtained xₖ. Subsequent, the output of this dense block is then forwarded to the primary transition layer, which is denoted as wᴛ. The ensuing tensor xᴛ is then concatenated with the half 1 tensor x₀’ earlier than finally being handed by way of the second transition layer wᴜ to acquire the ultimate output tensor xᴜ.
CSPDenseNet Implementation
Now let’s get even deeper into the CSPNet structure by implementing it from scratch. Though we are able to mainly apply the CSPNet construction to any spine, right here I’m going to try this on the DenseNet mannequin to match it with the illustrations and equations I confirmed you earlier. Determine 8 beneath shows what the whole DenseNet structure seems like. Simply do not forget that each single dense block on this structure initially follows the DenseNet construction in Determine 3a, and our goal right here is to switch all these dense blocks with CSPDenseNet block illustrated in Determine 3b.
The very first thing we do is to import the required modules and initialize the configurable parameters as proven in Codeblock 1. The GROWTH variable is the progress price parameter, which denotes the variety of characteristic maps produced by every bottleneck throughout the dense block. Subsequent, CHANNEL_POOLING is the parameter we use to regulate the habits of the channel-pooling mechanism in our first transition layer. Right here I set this parameter to 0.8, which means that we’ll shrink the variety of channels to 80% of its unique channel depend. The COMPRESSION parameter works equally to the CHANNEL_POOLING variable, but this one operates within the second transition layer. Lastly, right here we outline the REPEATS record, which is used to set the variety of bottleneck blocks we are going to initialize throughout the dense block of every stage.
# Codeblock 1
import torch
import torch.nn as nn
GROWTH = 12
CHANNEL_POOLING = 0.8
COMPRESSION = 0.5
REPEATS = [6, 12, 24, 16]Bottleneck Block Implementation
Beneath is the implementation of the bottleneck block to be positioned throughout the dense block. This Bottleneck class is precisely the identical because the one I utilized in my DenseNet article [3]. I immediately copy-pasted the code from there since we don’t want to change this half in any respect. Simply remember the fact that a bottleneck block contains a 1×1 convolution adopted by a 3×3 convolution.
# Codeblock 2
class Bottleneck(nn.Module):
def __init__(self, in_channels):
tremendous().__init__()
self.relu = nn.ReLU()
self.dropout = nn.Dropout(p=0.2)
self.bn0 = nn.BatchNorm2d(num_features=in_channels)
self.conv0 = nn.Conv2d(in_channels=in_channels,
out_channels=GROWTH*4,
kernel_size=1,
padding=0,
bias=False)
self.bn1 = nn.BatchNorm2d(num_features=GROWTH*4)
self.conv1 = nn.Conv2d(in_channels=GROWTH*4,
out_channels=GROWTH,
kernel_size=3,
padding=1,
bias=False)
def ahead(self, x):
print(f'originalt: {x.measurement()}')
out = self.dropout(self.conv0(self.relu(self.bn0(x))))
print(f'after conv0t: {out.measurement()}')
out = self.dropout(self.conv1(self.relu(self.bn1(out))))
print(f'after conv1t: {out.measurement()}')
concatenated = torch.cat((out, x), dim=1)
print(f'after concatt: {concatenated.measurement()}')
return concatenatedThe next testing code simulates the primary bottleneck block throughout the dense block. Keep in mind that the very first conv layer within the structure (the one with 7×7 kernel) produces 64 characteristic maps, however since within the case of CSPNet we solely need to course of half of them (the half 2 tensor), therefore right here we are going to check it with a tensor of 32 characteristic maps.
# Codeblock 3
bottleneck = Bottleneck(in_channels=32)
x = torch.randn(1, 32, 56, 56)
x = bottleneck(x)# Codeblock 3 Output
unique : torch.Dimension([1, 32, 56, 56])
after conv0 : torch.Dimension([1, 48, 56, 56])
after conv1 : torch.Dimension([1, 12, 56, 56])
after concat : torch.Dimension([1, 44, 56, 56])You may see within the ensuing output above that the variety of characteristic maps turns into 44 on the finish of the method, the place this quantity is obtained by including the enter channel depend and the expansion price, i.e., 32 + 12 = 44. Once more, you possibly can simply take a look at my DenseNet article [3] if you wish to get a greater understanding about this calculation.
Dense Block Implementation
Now to create a sequence of bottleneck blocks simply, we are able to simply wrap it contained in the DenseBlock class in Codeblock 4 beneath. In a while, we are able to simply specify the variety of bottleneck blocks to be stacked by way of the repeats parameter. Once more, this class can also be copy-pasted from my DenseNet article, so I’m not going to elucidate it any additional.
# Codeblock 4
class DenseBlock(nn.Module):
def __init__(self, in_channels, repeats):
tremendous().__init__()
self.bottlenecks = nn.ModuleList()
for i in vary(repeats):
current_in_channels = in_channels + i * GROWTH
self.bottlenecks.append(Bottleneck(in_channels=current_in_channels))
def ahead(self, x):
print(f'originalttt: {x.measurement()}')
for i, bottleneck in enumerate(self.bottlenecks):
x = bottleneck(x)
print(f'after bottleneck #{i}tt: {x.measurement()}')
return xWith the intention to test if our DenseBlock class works correctly, we are going to check it utilizing the Codeblock 5 beneath. Right here I’m making an attempt to simulate the half 2 tensor processed by the primary dense block, which accommodates a sequence of 6 bottleneck blocks.
# Codeblock 5
dense_block = DenseBlock(in_channels=32, repeats=6)
x = torch.randn(1, 32, 56, 56)
x = dense_block(x)And beneath is what the output seems like. Right here we are able to clearly see that every bottleneck block efficiently will increase the characteristic maps by 12.
# Codeblock 5 Output
unique : torch.Dimension([1, 32, 56, 56])
after bottleneck #0 : torch.Dimension([1, 44, 56, 56])
after bottleneck #1 : torch.Dimension([1, 56, 56, 56])
after bottleneck #2 : torch.Dimension([1, 68, 56, 56])
after bottleneck #3 : torch.Dimension([1, 80, 56, 56])
after bottleneck #4 : torch.Dimension([1, 92, 56, 56])
after bottleneck #5 : torch.Dimension([1, 104, 56, 56])First Transition
Keep in mind that the CSPDenseNet variant in Determine 3b makes use of two transition layers. On this part we’re going to focus on the primary transition layer, i.e., the one used to course of the tensor within the half 2 department. Right here we is not going to carry out spatial downsampling, which is the rationale why you don’t see any pooling layer throughout the __init__() methodology in Codeblock 6 beneath. As a substitute, right here we are going to solely carry out cross-channel pooling, which may be perceived as a typical pooling operation but is finished throughout the channel dimension. To implement it, we are able to merely use a 1×1 convolution (#(2)) and specify the variety of output channels we wish (#(1)). We are able to consider it like this: in a spatial downsampling course of, we are able to mainly do this through the use of both pooling or a strided convolution layer, which within the latter case it can mixture the pixel values with particular weightings from the native neighborhood. Within the case of cross-channel pooling, since we don’t have a particular PyTorch layer for that, we are able to merely change it with a pointwise convolution layer, which by doing so we are able to mainly mixture pixel values throughout the channel dimension.
# Codeblock 6
class FirstTransition(nn.Module):
def __init__(self, in_channels, out_channels):
tremendous().__init__()
self.bn = nn.BatchNorm2d(num_features=in_channels)
self.relu = nn.ReLU()
self.conv = nn.Conv2d(in_channels=in_channels,
out_channels=out_channels, #(1)
kernel_size=1, #(2)
padding=0,
bias=False)
self.dropout = nn.Dropout(p=0.2)
def ahead(self, x):
print(f'originaltt: {x.measurement()}')
out = self.dropout(self.conv(self.relu(self.bn(x))))
print(f'after first_transitiont: {out.measurement()}')
return outThe outcome given within the Codeblock 5 Output reveals that the half 2 tensor could have the form of 104×56×56 after being processed by the dense block. Thus, within the testing code beneath I’ll use this tensor form to simulate the primary transition layer inside that stage. To regulate the variety of output channels, we are able to merely multiply the enter channel depend with the CHANNEL_POOLING variable we initialized earlier as proven at line #(1) in Codeblock 7 beneath.
# Codeblock 7
first_transition = FirstTransition(in_channels=104,
out_channels=int(104*CHANNEL_POOLING)) #(1)
x = torch.randn(1, 104, 56, 56)
x = first_transition(x)Now because the code above is run, we are able to see that the variety of characteristic maps shrinks from 104 to 83 (80% of the unique).
# Codeblock 7 Output
unique : torch.Dimension([1, 104, 56, 56])
after first_transition : torch.Dimension([1, 83, 56, 56])Second Transition
The construction of the second transition layer is kind of a bit the identical as the primary one, besides that right here we even have a mean pooling layer with a stride of two to cut back the spatial dimension by half (#(1)).
# Codeblock 8
class SecondTransition(nn.Module):
def __init__(self, in_channels, out_channels):
tremendous().__init__()
self.bn = nn.BatchNorm2d(num_features=in_channels)
self.relu = nn.ReLU()
self.conv = nn.Conv2d(in_channels=in_channels,
out_channels=out_channels,
kernel_size=1,
padding=0,
bias=False)
self.dropout = nn.Dropout(p=0.2)
self.pool = nn.AvgPool2d(kernel_size=2, stride=2) #(1)
def ahead(self, x):
print(f'originaltt: {x.measurement()}')
out = self.pool(self.dropout(self.conv(self.relu(self.bn(x)))))
print(f'after second_transitiont: {out.measurement()}')
return outKeep in mind that the tensor coming into the second transition layer is a concatenation of the half 1 and the half 2 tensors. That is basically the rationale why within the testing code beneath I set this layer to simply accept 32 + 83 = 115 characteristic maps. Much like the primary transition layer, right here we multiply this variety of characteristic maps with the COMPRESSION variable (#(1)) to cut back the variety of channels even additional.
# Codeblock 9
second_transition = SecondTransition(in_channels=115,
out_channels=int(115*COMPRESSION)) #(1)
x = torch.randn(1, 115, 56, 56)
x = second_transition(x)Within the ensuing output beneath we are able to see that the spatial dimension halves because of the typical pooling layer. On the similar time, the variety of characteristic maps additionally decreases from 115 to 57 since we set the COMPRESSION parameter to 0.5.
# Codeblock 9 Output
unique : torch.Dimension([1, 115, 56, 56])
after second_transition : torch.Dimension([1, 57, 28, 28])The CSPDenseNet Mannequin
With all of the elements prepared, we are able to now construct the complete CSPDenseNet structure, which I break down in Codeblocks 10a, 10b, and 10c beneath. Let’s now give attention to the Codeblock 10a first, the place I initialize all of the layers based on the construction given in Determine 8. Right here you possibly can see at line #(1) that we initialize a 7×7 convolution layer, which acts because the enter layer of the community. This layer is then adopted by a maxpooling layer (#(2)). These two layers use the stride of two, which means that the spatial dimensions of the enter tensor will probably be lowered to one-fourth of its unique measurement.
# Codeblock 10a
class CSPDenseNet(nn.Module):
def __init__(self):
tremendous().__init__()
self.first_conv = nn.Conv2d(in_channels=3, #(1)
out_channels=64,
kernel_size=7,
stride=2,
padding=3,
bias=False)
self.first_pool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) #(2)
channel_count = 64
##### Stage 0
self.dense_block_0 = DenseBlock(in_channels=channel_count//2,
repeats=REPEATS[0])
self.first_transition_0 = FirstTransition(in_channels=(channel_count//2)+(REPEATS[0]*GROWTH),
out_channels=int(((channel_count//2)+(REPEATS[0]*GROWTH))*CHANNEL_POOLING))
channel_count = (channel_count - (channel_count//2)) + int(((channel_count//2)+(REPEATS[0]*GROWTH))*CHANNEL_POOLING)
self.second_transition_0 = SecondTransition(in_channels=channel_count,
out_channels=int(channel_count*COMPRESSION))
channel_count = int(channel_count*COMPRESSION)
#####
##### Stage 1
self.dense_block_1 = DenseBlock(in_channels=channel_count//2,
repeats=REPEATS[1])
self.first_transition_1 = FirstTransition(in_channels=(channel_count//2)+(REPEATS[1]*GROWTH),
out_channels=int(((channel_count//2)+(REPEATS[1]*GROWTH))*CHANNEL_POOLING))
channel_count = (channel_count - (channel_count//2)) + int(((channel_count//2)+(REPEATS[1]*GROWTH))*CHANNEL_POOLING)
self.second_transition_1 = SecondTransition(in_channels=channel_count,
out_channels=int(channel_count*COMPRESSION))
channel_count = int(channel_count*COMPRESSION)
#####
##### Stage 2
self.dense_block_2 = DenseBlock(in_channels=channel_count//2,
repeats=REPEATS[2])
self.first_transition_2 = FirstTransition(in_channels=(channel_count//2)+(REPEATS[2]*GROWTH),
out_channels=int(((channel_count//2)+(REPEATS[2]*GROWTH))*CHANNEL_POOLING))
channel_count = (channel_count - (channel_count//2)) + int(((channel_count//2)+(REPEATS[2]*GROWTH))*CHANNEL_POOLING)
self.second_transition_2 = SecondTransition(in_channels=channel_count,
out_channels=int(channel_count*COMPRESSION))
channel_count = int(channel_count*COMPRESSION)
#####
##### Stage 3
self.dense_block_3 = DenseBlock(in_channels=channel_count//2,
repeats=REPEATS[3])
self.first_transition_3 = FirstTransition(in_channels=(channel_count//2)+(REPEATS[3]*GROWTH),
out_channels=int(((channel_count//2)+(REPEATS[3]*GROWTH))*CHANNEL_POOLING))
channel_count = (channel_count - (channel_count//2)) + int(((channel_count//2)+(REPEATS[3]*GROWTH))*CHANNEL_POOLING)
#####
self.avgpool = nn.AdaptiveAvgPool2d(output_size=(1,1)) #(3)
self.fc = nn.Linear(in_features=channel_count, out_features=1000) #(4)Nonetheless with the above codeblock, right here I group the layers I initialize primarily based on the stage they belong to. Let’s now give attention to the half I seek advice from as Stage 0. Right here you possibly can see that we obtained a dense block (dense_block_0) and the primary transition layer (first_transition_0). These two elements are accountable to course of the half 2 tensor. Subsequent, we initialize the second transition layer (second_transition_0), which is used to course of the concatenation results of the half 1 and half 2 tensors. For the reason that channel depend is dynamic relying on the GROWTH, CHANNEL_POOLING, COMPRESSION, and REPEATS variables, we have to hold monitor of the channel depend after every step in order that the mannequin can adaptively modify itself based on these variables. We do the identical factor for all of the remaining phases, besides in Stage 3 we don’t initialize the second transition layer since at that time we received’t scale back the channels and the spatial dimension any additional. As a substitute, we are going to immediately go the concatenated half 1 and half 2 tensors to the typical pooling (#(3)) and the classification (#(4)) layers. And that ends our dialogue concerning the Codeblock 10a above.
Earlier than we get into the ahead() methodology, there’s one other perform we have to create: split_channels(). Because the title suggests, this perform, which is written in Codeblock 10b beneath, is used to separate a tensor into half 1 and half 2. The if-else assertion right here is used to test if the variety of channels is odd and even. The truth is, it might be very simple if the channel depend is an excellent quantity as we are able to simply divide them into two (#(4)). But when the channel depend is odd, we have to manually decide the scale of every half as seen at line #(1) and #(2) earlier than finally splitting them (#(3)).
# Codeblock 10b
def split_channels(self, x):
channel_count = x.measurement(1)
if channel_countpercent2 != 0:
split_size_2 = channel_count // 2 #(1)
split_size_1 = channel_count - split_size_2 #(2)
return torch.cut up(x, [split_size_1, split_size_2], dim=1) #(3)
else:
return torch.cut up(x, channel_count // 2, dim=1) #(4)As we now have completed defining the __init__() and the split_channel() strategies, we are able to now implement the ahead() methodology in Codeblock 10c beneath. Typically talking, what we do right here is just ahead the tensor sequentially. However now let’s take note of the half I seek advice from as Stage 0. Right here you possibly can see that after the tensor is handed by way of the first_pool layer (#(1)), we then cut up it into two utilizing the split_channels() perform we declared earlier (#(2)). From there, we now acquire the part1 and part2 tensors. We’ll go away the part1 tensor as is all the best way to the tip of the stage. In the meantime, for the part2 tensor, we are going to course of it with the dense block (#(3)) and the primary transition layer (#(4)). Subsequent, we concatenate the ensuing tensor with the part1 tensor to create the skip-connection (#(5)). After which, we lastly go it by way of the second transition layer (#(6)). The identical steps are repeated for all phases till we finally attain the output layer to make classification. Simply do not forget that the Stage 3 is kind of totally different as a result of right here we don’t have the second transition layer.
# Codeblock 10c
def ahead(self, x):
print(f'originalttt: {x.measurement()}')
x = self.first_conv(x)
print(f'after first_convtt: {x.measurement()}')
x = self.first_pool(x) #(1)
print(f'after first_pooltt: {x.measurement()}n')
##### Stage 0
part1, part2 = self.split_channels(x) #(2)
print(f'part1tttt: {part1.measurement()}')
print(f'part2tttt: {part2.measurement()}')
part2 = self.dense_block_0(part2) #(3)
print(f'part2 after dense block 0t: {part2.measurement()}')
part2 = self.first_transition_0(part2) #(4)
print(f'part2 after first trans 0t: {part2.measurement()}')
x = torch.cat((part1, part2), dim=1) #(5)
print(f'after concatenatett: {x.measurement()}')
x = self.second_transition_0(x) #(6)
print(f'after second transition 0t: {x.measurement()}n')
##### Stage 1
part1, part2 = self.split_channels(x)
print(f'part1tttt: {part1.measurement()}')
print(f'part2tttt: {part2.measurement()}')
part2 = self.dense_block_1(part2)
print(f'part2 after dense block 1t: {part2.measurement()}')
part2 = self.first_transition_1(part2)
print(f'part2 after first trans 1t: {part2.measurement()}')
x = torch.cat((part1, part2), dim=1)
print(f'after concatenatett: {x.measurement()}')
x = self.second_transition_1(x)
print(f'after second transition 1t: {x.measurement()}n')
##### Stage 2
part1, part2 = self.split_channels(x)
print(f'part1tttt: {part1.measurement()}')
print(f'part2tttt: {part2.measurement()}')
part2 = self.dense_block_2(part2)
print(f'part2 after dense block 2t: {part2.measurement()}')
part2 = self.first_transition_2(part2)
print(f'part2 after first trans 2t: {part2.measurement()}')
x = torch.cat((part1, part2), dim=1)
print(f'after concatenatett: {x.measurement()}')
x = self.second_transition_2(x)
print(f'after second transition 2t: {x.measurement()}n')
##### Stage 3
part1, part2 = self.split_channels(x)
print(f'part1tttt: {part1.measurement()}')
print(f'part2tttt: {part2.measurement()}')
part2 = self.dense_block_3(part2)
print(f'part2 after dense block 2t: {part2.measurement()}')
part2 = self.first_transition_3(part2)
print(f'part2 after first trans 2t: {part2.measurement()}')
x = torch.cat((part1, part2), dim=1)
print(f'after concatenatett: {x.measurement()}n')
x = self.avgpool(x)
print(f'after avgpoolttt: {x.measurement()}')
x = torch.flatten(x, start_dim=1)
print(f'after flattenttt: {x.measurement()}')
x = self.fc(x)
print(f'after fcttt: {x.measurement()}')
return xNow let’s check the CSPDenseNet class we simply created by operating the Codeblock 11 beneath. Right here I exploit a dummy tensor of form 3×224×224 to simulate a 224×224 RGB picture handed by way of the community.
# Codeblock 11
cspdensenet = CSPDenseNet()
x = torch.randn(1, 3, 224, 224)
x = cspdensenet(x)And beneath is what the output seems like. Right here you possibly can see that each time a tensor will get right into a community, our split_channels() methodology appropriately divides the tensor into two (#(1–2)). Then, the bottleneck block inside every stage additionally appropriately provides the variety of channels of the half 2 tensor by 12 earlier than finally being handed by way of the primary transition layer. The primary transition layer itself efficiently reduces the variety of channels by 20% as seen at line #(3), simulating the cross-channel pooling mechanism. Afterwards, the ensuing tensor is then concatenated with the tensor from half 1 (#(4)) and handed by way of the second transition layer (#(5)) to additional scale back the variety of channels and halve the spatial dimension. We do the identical factor for all phases till finally we obtained the 1000-class prediction.
# Codeblock 11 Output
unique : torch.Dimension([1, 3, 224, 224])
after first_conv : torch.Dimension([1, 64, 112, 112])
after first_pool : torch.Dimension([1, 64, 56, 56])
part1 : torch.Dimension([1, 32, 56, 56]) #(1)
part2 : torch.Dimension([1, 32, 56, 56]) #(2)
after bottleneck #0 : torch.Dimension([1, 44, 56, 56])
after bottleneck #1 : torch.Dimension([1, 56, 56, 56])
after bottleneck #2 : torch.Dimension([1, 68, 56, 56])
after bottleneck #3 : torch.Dimension([1, 80, 56, 56])
after bottleneck #4 : torch.Dimension([1, 92, 56, 56])
after bottleneck #5 : torch.Dimension([1, 104, 56, 56])
part2 after dense block 0 : torch.Dimension([1, 104, 56, 56])
part2 after first trans 0 : torch.Dimension([1, 83, 56, 56]) #(3)
after concatenate : torch.Dimension([1, 115, 56, 56]) #(4)
after second transition 0 : torch.Dimension([1, 57, 28, 28]) #(5)
part1 : torch.Dimension([1, 29, 28, 28])
part2 : torch.Dimension([1, 28, 28, 28])
after bottleneck #0 : torch.Dimension([1, 40, 28, 28])
after bottleneck #1 : torch.Dimension([1, 52, 28, 28])
after bottleneck #2 : torch.Dimension([1, 64, 28, 28])
after bottleneck #3 : torch.Dimension([1, 76, 28, 28])
after bottleneck #4 : torch.Dimension([1, 88, 28, 28])
after bottleneck #5 : torch.Dimension([1, 100, 28, 28])
after bottleneck #6 : torch.Dimension([1, 112, 28, 28])
after bottleneck #7 : torch.Dimension([1, 124, 28, 28])
after bottleneck #8 : torch.Dimension([1, 136, 28, 28])
after bottleneck #9 : torch.Dimension([1, 148, 28, 28])
after bottleneck #10 : torch.Dimension([1, 160, 28, 28])
after bottleneck #11 : torch.Dimension([1, 172, 28, 28])
part2 after dense block 1 : torch.Dimension([1, 172, 28, 28])
part2 after first trans 1 : torch.Dimension([1, 137, 28, 28])
after concatenate : torch.Dimension([1, 166, 28, 28])
after second transition 1 : torch.Dimension([1, 83, 14, 14])
part1 : torch.Dimension([1, 42, 14, 14])
part2 : torch.Dimension([1, 41, 14, 14])
after bottleneck #0 : torch.Dimension([1, 53, 14, 14])
after bottleneck #1 : torch.Dimension([1, 65, 14, 14])
after bottleneck #2 : torch.Dimension([1, 77, 14, 14])
after bottleneck #3 : torch.Dimension([1, 89, 14, 14])
after bottleneck #4 : torch.Dimension([1, 101, 14, 14])
after bottleneck #5 : torch.Dimension([1, 113, 14, 14])
after bottleneck #6 : torch.Dimension([1, 125, 14, 14])
after bottleneck #7 : torch.Dimension([1, 137, 14, 14])
after bottleneck #8 : torch.Dimension([1, 149, 14, 14])
after bottleneck #9 : torch.Dimension([1, 161, 14, 14])
after bottleneck #10 : torch.Dimension([1, 173, 14, 14])
after bottleneck #11 : torch.Dimension([1, 185, 14, 14])
after bottleneck #12 : torch.Dimension([1, 197, 14, 14])
after bottleneck #13 : torch.Dimension([1, 209, 14, 14])
after bottleneck #14 : torch.Dimension([1, 221, 14, 14])
after bottleneck #15 : torch.Dimension([1, 233, 14, 14])
after bottleneck #16 : torch.Dimension([1, 245, 14, 14])
after bottleneck #17 : torch.Dimension([1, 257, 14, 14])
after bottleneck #18 : torch.Dimension([1, 269, 14, 14])
after bottleneck #19 : torch.Dimension([1, 281, 14, 14])
after bottleneck #20 : torch.Dimension([1, 293, 14, 14])
after bottleneck #21 : torch.Dimension([1, 305, 14, 14])
after bottleneck #22 : torch.Dimension([1, 317, 14, 14])
after bottleneck #23 : torch.Dimension([1, 329, 14, 14])
part2 after dense block 2 : torch.Dimension([1, 329, 14, 14])
part2 after first trans 2 : torch.Dimension([1, 263, 14, 14])
after concatenate : torch.Dimension([1, 305, 14, 14])
after second transition 2 : torch.Dimension([1, 152, 7, 7])
part1 : torch.Dimension([1, 76, 7, 7])
part2 : torch.Dimension([1, 76, 7, 7])
after bottleneck #0 : torch.Dimension([1, 88, 7, 7])
after bottleneck #1 : torch.Dimension([1, 100, 7, 7])
after bottleneck #2 : torch.Dimension([1, 112, 7, 7])
after bottleneck #3 : torch.Dimension([1, 124, 7, 7])
after bottleneck #4 : torch.Dimension([1, 136, 7, 7])
after bottleneck #5 : torch.Dimension([1, 148, 7, 7])
after bottleneck #6 : torch.Dimension([1, 160, 7, 7])
after bottleneck #7 : torch.Dimension([1, 172, 7, 7])
after bottleneck #8 : torch.Dimension([1, 184, 7, 7])
after bottleneck #9 : torch.Dimension([1, 196, 7, 7])
after bottleneck #10 : torch.Dimension([1, 208, 7, 7])
after bottleneck #11 : torch.Dimension([1, 220, 7, 7])
after bottleneck #12 : torch.Dimension([1, 232, 7, 7])
after bottleneck #13 : torch.Dimension([1, 244, 7, 7])
after bottleneck #14 : torch.Dimension([1, 256, 7, 7])
after bottleneck #15 : torch.Dimension([1, 268, 7, 7])
part2 after dense block 2 : torch.Dimension([1, 268, 7, 7])
part2 after first trans 2 : torch.Dimension([1, 214, 7, 7])
after concatenate : torch.Dimension([1, 290, 7, 7])
after avgpool : torch.Dimension([1, 290, 1, 1])
after flatten : torch.Dimension([1, 290])
after fc : torch.Dimension([1, 1000])Ending
And that’s it! We now have efficiently discovered CSPNet and carried out it on DenseNet spine. As I’ve talked about earlier, we are able to truly use the thought of CSPNet to enhance the efficiency of another spine fashions reminiscent of ResNet or ResNeXt. So right here I problem you to implement CSPNet on these fashions from scratch.
To be sincere I can not verify that my implementation is 100% right for the reason that official GitHub repo [4] of the paper doesn’t present the PyTorch implementation — however that’s not less than the whole lot I perceive from the manuscript. Please let me know in case you discover any mistake within the code or in my explanations. Thanks for studying, and see you once more in my subsequent article. Bye!
Btw you can even discover the code used on this article on my GitHub repo [5].
References
[1] Chien-Yao Wang et al. CSPnet: A New Spine That Can Improve Studying Functionality of CNN. Arxiv. https://arxiv.org/abs/1911.11929 [Accessed October 1, 2025].
[2] Gao Huang et al. Densely Related Convolutional Networks. Arxiv. https://arxiv.org/abs/1608.06993 [Accessed September 18, 2025].
[3] Muhammad Ardi. DenseNet Paper Walkthrough: All Related. In direction of Knowledge Science. https://towardsdatascience.com/densenet-paper-walkthrough-all-connected/ [Accessed April 26, 2026].
[4] WongKinYiu. CrossStagePartialNetworks. GitHub. https://github.com/WongKinYiu/CrossStagePartialNetworks [Accessed October 1, 2025].
[5] MuhammadArdiPutra. CSPNet. GitHub. https://github.com/MuhammadArdiPutra/medium_articles/blob/fundamental/DenseNet.ipynb [Accessed October 1, 2025].
