Consider emptying TensorKit cache after each CTMRG iteration?

[Yue-Zhengyuan]

I find that calling TensorKit.empty_globalcaches!() can significantly reduce memory usage when dealing with product of several symmetries: just inserting it between these two lines.

PEPSKit.jl/src/algorithms/ctmrg/ctmrg.jl

Lines 114 to 115 in 7ceb4a3

	env, info = ctmrg_iteration(network, env, alg)
	η, CS, TS = calc_convergence(env, CS, TS)

In my case I was contracting a PEPS with fZ2 x U(1) x U(1) symmetry, D = 8, using environment chi = 32. I use the sequential CTMRG with fullinfinite for the projectors. I was also restricting the cache sizes:

resize!(TensorKit.GLOBAL_FSTRANSPOSE_CACHE; maxsize = 400)
resize!(TensorKit.GLOBAL_FSBRAID_CACHE; maxsize = 400)
resize!(TensorKit.GLOBAL_FUSIONBLOCKSTRUCTURE_CACHE; maxsize = 400)
resize!(TensorKit.GLOBAL_TREEBRAIDER_CACHE; maxsize = 400)
resize!(TensorKit.GLOBAL_TREEPERMUTER_CACHE; maxsize = 400)
resize!(TensorKit.GLOBAL_TREETRANSPOSER_CACHE; maxsize = 400)

Using current master branch of PEPSKit, memory usage can go up to 24 GB, which is even more than the amount used fZ2 x U(1). But if I clear cache after each CTMRG iteration, memory usage can be reduced by half, though this seems to have some negative effect on the speed of CTMRG (I haven't been able to do a controlled test due to limited resources).

IIRC, @leburgel was also bothered by the memory issue and may find this cache clearing helpful.

[lkdvos]

I think what might be a slightly better alternative that could have the same effect is to check how much of each cache is used during a single CTMRG iteration, and then fix the maximal size to that. Since the caches are LRU, they really should handle these things automatically, and the trick is in finding good maximal sizes for them.
What is a bit annoying though is that the different caches somewhat depend on eachother, in the sense that the FSTRANSPOSE_CACHE (level of fusion trees) contains data that is needed for constructing the TREETRANSPOSER_CACHE data (level of tensor structure), which basically means that there are two possible cases:

1. The tensor structure doesn't change between iterations, so we don't compute new TREEPERMUTER data if we keep that in the cache, therefore we can empty the FSTRANSPOSE_CACHE since this is no longer needed.
2. The structure changes, but most of the charges remain the same. In this case, the TREEPERMUTER_CACHE is mostly useless, since it is unlikely it can be used again, but the FSTRANSPOSE_CACHE ensures that we can still rapidly construct the new data required to permute the tensors.

I think I recall adding a method to print cache information to TensorKit a while ago, it might be useful to just check where all of this memory is really going? I will try and think a bit more about possible ways to have better or more advanced cache systems, but I don't think emptying them after each iteration really is an option, since in that case we might as well turn them off to begin with, because then the reuse is quite limited

[leburgel]

I did some digging, and for my particular use case (which is a product of 3 symmetries, one of which is non-Abelian) by far the most memory is in the FUSIONBLOCKSTRUCTURE_CACHE. All the others are actually quite reasonably sized in comparison (in that they all have a memory size that is at least an order of magnitude smaller). It's a bit of a jump to assume this is generic behavior, but it does seem logical that the amount of memory taken up by a single FUSIONBLOCKSTRUCTURE_CACHE increases a lot when more symmetries are added as a product.

Because of this scaling, setting a maximum cache size in terms of number of entries is actually not very useful in practice, since it's very hard (for me at least) to predict how this will translate to memory usage. It would be very useful if we could set maximum memory sizes for the caches instead, at least for the cases where I have really run into trouble.

It would also be great if there were some possible memory savings left in how the fusion block structure is cached that are usually just not necessary, but could reduce the memory pressure for product symmetries. But I have no idea if there's anything to do about this conceptually.