Bug? Always zero gradient for model.scale

[vadimkantorov]

Hi @ines-chami!

At https://github.com/HazyResearch/HypHC/blob/master/model/hyphc.py#L42 :

init_size=1e-3        # in config.py also "init_size": 1e-3
max_scale=1. - 1e-3   # in config.py also "max_scale": 1 - 1e-3
self.scale = nn.Parameter(torch.Tensor([init_size]), requires_grad=True)

min_scale = 1e-2 #self.init_size
max_scale = self.max_scale
return F.normalize(embeddings, p=2, dim=1) * self.scale.clamp_min(min_scale).clamp_max(max_scale)

So self.scale (initialized always to init_size = 1e-3) is always outside the clamp range (min_scale = 1e-2 and max_scale = 1 - 1e-3), and so always receives zero gradient.

Is it expected / by design or was it some debug setting min_scale = 1e-2 which by mistake was not removed?