Prob when score a DenseNet

[mm0806son]

We used the DenseNet model cloned from Kuangliu. In this code the pooling layer and ReLu layer are defined by:

out = F.avg_pool2d(F.relu(self.bn(out)), 4)

Instead of using:

def __init__():
    ...
    self.avg_pool2d = nn.AvgPool2d(4)
    self.Relu = nn.ReLU()
def forward(self, x):
    ...
    out = self.Relu(out)
    out = self.avg_pool2d(out)
    ...

In the profile.py, as the code:

def add_hooks(m):
    if isinstance(m, nn.Conv2d):
        m.register_forward_hook(count_conv2d)
    elif isinstance(m, nn.BatchNorm2d):
        m.register_forward_hook(count_bn2d)
    elif isinstance(m, nn.ReLU):
        m.register_forward_hook(count_relu)
    elif isinstance(m, (nn.AvgPool2d)):
        m.register_forward_hook(count_avgpool)
    elif isinstance(m, nn.Linear):
        m.register_forward_hook(count_linear)
    elif isinstance(m, nn.Sequential):
        m.register_forward_hook(count_sequential)
    else:
        print("Not implemented for ", m)

So the scoring system will not count the pooling layers and ReLu layers and then shows a fake score slightly better than the real one.

In practise, I here use our best model as an example. By changing replace this line of original code by a nn.module version:

# out = F.avg_pool2d(F.relu(self.bn(out)), 4)
out = self.bn(out)
out = self.relu(out)
out = self.avg_pool2d(out)

I got a different number of operations as:

# before
Flops: 22675830.0, Params: 59573.0
Score flops: 0.027177419494021596 Score Params: 0.010662824878051312
Final score: 0.03784024437207291

#after
Flops: 22678390.0, Params: 59573.0
Score flops: 0.027180487703383927 Score Params: 0.010662824878051312
Final score: 0.03784331258143524

We can also notice that before the change, pooling and relu layers are not printed.

If a "competitor" didn't notice that, it would be (slightly, if not intented) unfair to others. So I think maybe it would be better to optimize profile.py to take this issue into account or give a standard implementation of models to everyone.