Pad 134

[will-HPE]

Tensorboard and GPU kernel build fixes (PAD-91 and PAD-134)

• fix: bug fix

1. The latest Tensorboard version brought down has issues so we set the required version to one known to work (PAD-91); and
2. During GPU kernel build the cleanup fails throwing an error (PAD-134) we set an environmental variable prevent the cleanup of directories that are not empty.

Checklist

• [✓ ] Bump VERSION to make the pushed images are tagged with the right version.
• Licenses should be included for new code which was copied and/or modified from any external code.
• Test the images by running the test bumpenvs procedure in the determined repo. See README.

[karlonw]

looks good to me

[ioga]

hold up, this can't be right, this is a tensorboard version for tensorflow v1 from 2019.
can we have a version that is more modern?

[will-HPE]

We can test a different version but 2.10.1 fails on end-to-end testing and was imortalized in jira PAD-91:
https://hpe-aiatscale.atlassian.net/browse/PAD-91

103_run_mlde_validation_suite_against_rocm_on_grenoble/determined-...
tests/nightly/test_pytorch2.py::test_pytorch2_hf_language_modeling_distributed FAILED [100%]

The reason for the failure is:

ImportError: TensorBoard logging requires TensorBoard version 1.15 or above

[ioga]

I have a strong opinion we cannot pin 1.15 here because it's 5 years old and likely to conflict with other dependencies and have CVEs.
2.10.1 is also technically above 1.15...

[ioga]

we've just released 0.30.0; if it'll land (and get into bumpenvs) by EOD today, you can keep it at 0.30.1. otherwise it'll probably be 0.30.2