Training bottleneck #45
Description
Large bottlenecks when training. I trained for 12 hours with cProfile running and got this as the top cumulative time functions. It shows that Batch.fill takes up 85% of training time!
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.002 0.002 39087.736 39087.736 Train.py:1(<module>)
1 3.653 3.653 39086.782 39086.782 Train.py:16(main)
145058 1.010 0.000 35739.204 0.246 Train.py:44(run_net)
145058 129.018 0.001 33142.496 0.228 Batch.py:34(fill)
10049055 21.451 0.000 22053.800 0.002 Data.py:39(get_data)
10049055 1976.519 0.000 22032.350 0.002 Segment_Data.py:275(get_data)
64993847 9822.780 0.000 15071.341 0.000 dataset.py:397(__getitem__)
9283661 755.755 0.000 10920.047 0.001 Batch.py:49(data_into_batch)
114014974/112709452 5766.228 0.000 5781.245 0.000 {torch._C.cat}
129980296 2802.223 0.000 4964.670 0.000 group.py:160(__getitem__)
145057 2.188 0.000 3308.168 0.023 Batch.py:111(backward)
65565909 579.216 0.000 3170.871 0.000 _utils.py:37(_cuda)
145057 261.668 0.002 2595.698 0.018 Batch.py:97(forward)
64993847 277.916 0.000 2158.638 0.000 selections.py:27(select)
145057 12.529 0.000 1901.331 0.013 clip_grad.py:2(clip_grad_norm)
7542964 1797.248 0.000 1797.248 0.000 {method 'norm' of 'torch._C.CudaFloatTensorBase' objects}
65420851 1795.412 0.000 1795.412 0.000 {method 'copy_' of 'torch._C.CudaFloatTensorBase' objects}
64993847 1425.956 0.000 1740.737 0.000 dataset.py:313(__init__)
56137097 229.232 0.000 1557.593 0.000 __init__.py:267(type)
259975388 1315.414 0.000 1348.605 0.000 dataset.py:217(shape)
64993847 392.916 0.000 1277.835 0.000 selections.py:250(__getitem__)
10009001/290115 39.002 0.000 1142.344 0.004 module.py:205(__call__)
56137099 285.072 0.000 1116.109 0.000 _utils.py:5(_type)
145058 2.763 0.000 1107.541 0.008 SqueezeNet.py:76(forward)
435174 4.181 0.000 1074.084 0.002 container.py:62(forward)
27850982 50.400 0.000 1012.754 0.000 tensor.py:37(float)
1160464 13.646 0.000 965.766 0.001 SqueezeNet.py:25(forward)
64993847 125.658 0.000 907.826 0.000 fromnumeric.py:1837(product)
145057 1.007 0.000 895.441 0.006 variable.py:116(backward)
145057 892.171 0.006 892.171 0.006 {method 'run_backward' of 'torch._C._EngineBase' objects}
74279245 829.433 0.000 829.433 0.000 {method 'reduce' of 'numpy.ufunc' objects}
64993847 288.611 0.000 794.311 0.000 selections.py:429(_handle_simple)
18567321 724.864 0.000 724.864 0.000 {method 'copy_' of 'torch._C.CudaDoubleTensorBase' objects}
1305522 8.819 0.000 622.831 0.000 variable.py:839(cat)
27850944 30.437 0.000 619.597 0.000 tensor.py:29(cpu)
1305522 3.550 0.000 612.082 0.000 tensor.py:308(forward)
64993847 186.837 0.000 539.964 0.000 selections.py:244(__init__)
145057 70.703 0.000 509.208 0.004 adadelta.py:27(step)
64993847 320.490 0.000 325.345 0.000 selections.py:147(__init__)
27850944 311.320 0.000 311.320 0.000 {method 'copy_' of 'torch._C.FloatTensorBase' objects}
1281151179/1281151177 261.754 0.000 288.256 0.000 {isinstance}
64993847 249.225 0.000 281.225 0.000 filters.py:207(get_filters)
9283661 279.629 0.000 279.629 0.000 {method 'copy_' of 'torch._C.CudaByteTensorBase' objects}
64993847 177.023 0.000 269.627 0.000 selections.py:406(_expand_ellipsis)
64993847 268.112 0.000 268.112 0.000 base.py:81(is_empty_dataspace)
27850981 20.463 0.000 259.458 0.000 tensor.py:312(__div__)
27850981 238.995 0.000 238.995 0.000 {method 'div' of 'torch._C.CudaFloatTensorBase' objects}
18567374 220.069 0.000 220.069 0.000 {method 'zero_' of 'torch._C.FloatTensorBase' objects}
9283648 6.272 0.000 208.434 0.000 {method 'mean' of 'numpy.ndarray' objects}
18567309 13.199 0.000 205.147 0.000 tensor.py:273(__sub__)
3771508 11.772 0.000 202.183 0.000 conv.py:235(forward)
9283648 101.293 0.000 202.162 0.000 _methods.py:53(_mean)
64986860 159.096 0.000 192.368 0.000 group.py:36(__init__)
18567309 191.948 0.000 191.948 0.000 {method 'sub' of 'torch._C.CudaFloatTensorBase' objects}