Test scores

[rabbitholejourneys]

Thank you for sharing the work!

I tried running the code and training worked fine (high accuracy for both edges and points).
However, testing showed bad results. For the default batch_size of 1 it crashed (it seems like in a lot of cases, no points or edges can be detected and there is an empty tensor error). For a batch_size of 64 it works, but I get the following metrics:

>python train.py --gpu=2 --extra_tag pts66 
epochs: 100%|█| 90/90 [4:17:20<00:00, 171.56s/it, pts_acc=0.956, edge_acc=1, pts_cls_loss=0.209, pts_offset_loss=0.122, pts_loss=0.33, refine_offset_loss=0.0202, refine_loss=0.0202, edge_cls_loss=0.00586, edge_loss=0.00586
>python test.py --gpu=2 --test_tag pts66 --batch_size 64
test: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [00:13<00:00,  1.90it/s]
2024-10-07 14:31:02,548   INFO  pts_recall: 0.093226
2024-10-07 14:31:02,549   INFO  pts_precision: 0.110749
2024-10-07 14:31:02,549   INFO  pts_bias: 0.288388, 0.315627, 0.224667
2024-10-07 14:31:02,549   INFO  edge_recall: 0.326824
2024-10-07 14:31:02,549   INFO  edge_precision: 0.365887

Is this to be expected or do you have any ideas how to fix it? What values should be reached here?

I tried with pytorch 1.11 and 2.4, and adapted the cuda code from THC to ATen, which might be a breaking point (https://github.com/rabbitholejourneys/Point2Roof/tree/pytorch24).
Furthermore, I tried to run the testing in training mode as training worked fine, but the results did not get better.

[qiyang-git]

Thank you for sharing the work!

I tried running the code and training worked fine (high accuracy for both edges and points). However, testing showed bad results. For the default batch_size of 1 it crashed (it seems like in a lot of cases, no points or edges can be detected and there is an empty tensor error). For a batch_size of 64 it works, but I get the following metrics:

>python train.py --gpu=2 --extra_tag pts66 
epochs: 100%|█| 90/90 [4:17:20<00:00, 171.56s/it, pts_acc=0.956, edge_acc=1, pts_cls_loss=0.209, pts_offset_loss=0.122, pts_loss=0.33, refine_offset_loss=0.0202, refine_loss=0.0202, edge_cls_loss=0.00586, edge_loss=0.00586
>python test.py --gpu=2 --test_tag pts66 --batch_size 64
test: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [00:13<00:00,  1.90it/s]
2024-10-07 14:31:02,548   INFO  pts_recall: 0.093226
2024-10-07 14:31:02,549   INFO  pts_precision: 0.110749
2024-10-07 14:31:02,549   INFO  pts_bias: 0.288388, 0.315627, 0.224667
2024-10-07 14:31:02,549   INFO  edge_recall: 0.326824
2024-10-07 14:31:02,549   INFO  edge_precision: 0.365887

Is this to be expected or do you have any ideas how to fix it? What values should be reached here?

I tried with pytorch 1.11 and 2.4, and adapted the cuda code from THC to ATen, which might be a breaking point (https://github.com/rabbitholejourneys/Point2Roof/tree/pytorch24). Furthermore, I tried to run the testing in training mode as training worked fine, but the results did not get better.

Hi, Rabbit, did you have the C++ compile on the installation part?

[rabbitholejourneys]

Hi, Rabbit, did you have the C++ compile on the installation part?

Hi,
The following is what I did for python 3.9 with PyTorch 1.11:

conda create -p ~/condas/p2rpy39/ python=3.9
conda activate ~/condas/p2rpy39/
conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 numpy=1.20 -c pytorch
conda install -c conda-forge cudatoolkit-dev=11.3 --solver=classic
conda install -c conda-forge scikit-learn scipy tqdm easydict pyaml --solver=classic
conda install -c conda-forge gxx_linux-64=10.*
python -m pip install ./pc_util/
CUDA_VISIBLE_DEVICES=2 python train.py --gpu=2 --extra_tag pts66 --batch-size=5
CUDA_VISIBLE_DEVICES=2 python test.py --gpu=2 --test_tag pts66 --batch_size=1

And for pytorch 2.4:

conda create -p  ~/condas/p2r/ python=3.10
conda activate ~/condas/p2r
conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia
conda install -c conda-forge scikit-learn scipy tqdm easydict --solver=classic
python -m pip install ~/repos/Point2Roof/pc_util

I had to adapt the code before it compiled (see my fork above).
If you run the training and testing, I would be very interested in your results, because my test scores suggest that it the model does not really generalise, even though training seemed fine. I don know why or how to fix it.

[qiyang-git]

Hi, Rabbit, did you have the C++ compile on the installation part?

Hi, The following is what I did for python 3.9 with PyTorch 1.11:

conda create -p ~/condas/p2rpy39/ python=3.9
conda activate ~/condas/p2rpy39/
conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 numpy=1.20 -c pytorch
conda install -c conda-forge cudatoolkit-dev=11.3 --solver=classic
conda install -c conda-forge scikit-learn scipy tqdm easydict pyaml --solver=classic
conda install -c conda-forge gxx_linux-64=10.*
python -m pip install ./pc_util/
CUDA_VISIBLE_DEVICES=2 python train.py --gpu=2 --extra_tag pts66 --batch-size=5
CUDA_VISIBLE_DEVICES=2 python test.py --gpu=2 --test_tag pts66 --batch_size=1

And for pytorch 2.4:

conda create -p  ~/condas/p2r/ python=3.10
conda activate ~/condas/p2r
conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia
conda install -c conda-forge scikit-learn scipy tqdm easydict --solver=classic
python -m pip install ~/repos/Point2Roof/pc_util

I had to adapt the code before it compiled (see my fork above). If you run the training and testing, I would be very interested in your results, because my test scores suggest that it the model does not really generalise, even though training seemed fine. I don know why or how to fix it.

Thank you so much, Rabbit. I will try your code and let you know my result.