Output looks Poor

[AntonyEddard]

I have implemented EVER in my computer, but my output looks poor than a 3DGS output. What might be the issue ? I have implemented the docker file.. because I was not able to run the normal files. Can you help me with this case..

[half-potato]

Can you post some information? Input images, output images, dataset, training logs, etc.

[AntonyEddard]

Input images are the colmap undistorted output, we had 711 input images.
we are using 360 images converted into perspective images (711 images) and use it in colmap.

this is the log. It has slang error.

== CUDA ==

CUDA Version 12.2.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

/ever_training2/ever/splinetracers/slang/optix-intrinsics.slang(509): warning 41010: non-void function does not return in all cases
uint get_payload(uint i) {
^~~~~~~~~~~

/ever_training2/ever/splinetracers/slang/optix-intrinsics.slang(509): warning 41010: non-void function does not return in all cases
uint get_payload(uint i) {
^~~~~~~~~~~

/ever_training2/ever/splinetracers/slang/sh-half.slang(21): warning 30081: implicit conversion from 'float' to 'half' is not recommended
1.0925484305920792f, -1.0925484305920792f, 0.31539156525252005f, -1.0925484305920792f,
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(21): warning 30081: implicit conversion from 'float' to 'half' is not recommended
1.0925484305920792f, -1.0925484305920792f, 0.31539156525252005f, -1.0925484305920792f,
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(21): warning 30081: implicit conversion from 'float' to 'half' is not recommended
1.0925484305920792f, -1.0925484305920792f, 0.31539156525252005f, -1.0925484305920792f,
^~~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(21): warning 30081: implicit conversion from 'float' to 'half' is not recommended
1.0925484305920792f, -1.0925484305920792f, 0.31539156525252005f, -1.0925484305920792f,
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(22): warning 30081: implicit conversion from 'float' to 'half' is not recommended
0.5462742152960396f
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(25): warning 30081: implicit conversion from 'float' to 'half' is not recommended
-0.5900435899266435f, 2.890611442640554f, -0.4570457994644658f, 0.3731763325901154f,
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(25): warning 30081: implicit conversion from 'float' to 'half' is not recommended
-0.5900435899266435f, 2.890611442640554f, -0.4570457994644658f, 0.3731763325901154f,
^~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(25): warning 30081: implicit conversion from 'float' to 'half' is not recommended
-0.5900435899266435f, 2.890611442640554f, -0.4570457994644658f, 0.3731763325901154f,
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(25): warning 30081: implicit conversion from 'float' to 'half' is not recommended
-0.5900435899266435f, 2.890611442640554f, -0.4570457994644658f, 0.3731763325901154f,
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(26): warning 30081: implicit conversion from 'float' to 'half' is not recommended
-0.4570457994644658f, 1.445305721320277f, -0.5900435899266435f
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(26): warning 30081: implicit conversion from 'float' to 'half' is not recommended
-0.4570457994644658f, 1.445305721320277f, -0.5900435899266435f
^~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(26): warning 30081: implicit conversion from 'float' to 'half' is not recommended
-0.4570457994644658f, 1.445305721320277f, -0.5900435899266435f
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(18): warning 30081: implicit conversion from 'float' to 'half' is not recommended
static const half SH_C0 = 0.28209479177387814f;
^~~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(19): warning 30081: implicit conversion from 'float' to 'half' is not recommended
static const half SH_C1 = 0.4886025119029199f;
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(50): warning 30081: implicit conversion from 'vector<float,3>' to 'vector<half,3>' is not recommended
return SH_C0 * feat.f0 + 0.5f;
^
/ever_training2/ever/splinetracers/slang/sh-half.slang(71): warning 30081: implicit conversion from 'vector<float,3>' to 'vector<half,3>' is not recommended
SH_C2[3] * xz * feat.f7 + SH_C2[4] * (xx - yy) * feat.f8;
^
/ever_training2/ever/splinetracers/slang/sh-half.slang(86): warning 30081: implicit conversion from 'vector<float,3>' to 'vector<half,3>' is not recommended
SH_C3[5] * z * (xx - yy) * feat.f14 +
^

/ever_training2/ever/splinetracers/slang/sh-half.slang(21): warning 30081: implicit conversion from 'float' to 'half' is not recommended
1.0925484305920792f, -1.0925484305920792f, 0.31539156525252005f, -1.0925484305920792f,
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(21): warning 30081: implicit conversion from 'float' to 'half' is not recommended
1.0925484305920792f, -1.0925484305920792f, 0.31539156525252005f, -1.0925484305920792f,
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(21): warning 30081: implicit conversion from 'float' to 'half' is not recommended
1.0925484305920792f, -1.0925484305920792f, 0.31539156525252005f, -1.0925484305920792f,
^~~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(21): warning 30081: implicit conversion from 'float' to 'half' is not recommended
1.0925484305920792f, -1.0925484305920792f, 0.31539156525252005f, -1.0925484305920792f,
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(22): warning 30081: implicit conversion from 'float' to 'half' is not recommended
0.5462742152960396f
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(25): warning 30081: implicit conversion from 'float' to 'half' is not recommended
-0.5900435899266435f, 2.890611442640554f, -0.4570457994644658f, 0.3731763325901154f,
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(25): warning 30081: implicit conversion from 'float' to 'half' is not recommended
-0.5900435899266435f, 2.890611442640554f, -0.4570457994644658f, 0.3731763325901154f,
^~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(25): warning 30081: implicit conversion from 'float' to 'half' is not recommended
-0.5900435899266435f, 2.890611442640554f, -0.4570457994644658f, 0.3731763325901154f,
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(25): warning 30081: implicit conversion from 'float' to 'half' is not recommended
-0.5900435899266435f, 2.890611442640554f, -0.4570457994644658f, 0.3731763325901154f,
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(26): warning 30081: implicit conversion from 'float' to 'half' is not recommended
-0.4570457994644658f, 1.445305721320277f, -0.5900435899266435f
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(26): warning 30081: implicit conversion from 'float' to 'half' is not recommended
-0.4570457994644658f, 1.445305721320277f, -0.5900435899266435f
^~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(26): warning 30081: implicit conversion from 'float' to 'half' is not recommended
-0.4570457994644658f, 1.445305721320277f, -0.5900435899266435f
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(18): warning 30081: implicit conversion from 'float' to 'half' is not recommended
static const half SH_C0 = 0.28209479177387814f;
^~~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(19): warning 30081: implicit conversion from 'float' to 'half' is not recommended
static const half SH_C1 = 0.4886025119029199f;
^~~~~~~~~~~~~~~~~~~
/ever_training2/ever/splinetracers/slang/sh-half.slang(50): warning 30081: implicit conversion from 'vector<float,3>' to 'vector<half,3>' is not recommended
return SH_C0 * feat.f0 + 0.5f;
^
/ever_training2/ever/splinetracers/slang/sh-half.slang(71): warning 30081: implicit conversion from 'vector<float,3>' to 'vector<half,3>' is not recommended
SH_C2[3] * xz * feat.f7 + SH_C2[4] * (xx - yy) * feat.f8;
^
/ever_training2/ever/splinetracers/slang/sh-half.slang(86): warning 30081: implicit conversion from 'vector<float,3>' to 'vector<half,3>' is not recommended
SH_C3[5] * z * (xx - yy) * feat.f14 +
^

Optimizing /data/output
Output folder: /data/output [07/03 12:59:42]
Tensorboard not available: not logging progress [07/03 12:59:42]
Training progress: 0%| | 0/30000 [00:00<?, ?it/s] [07/03 12:59:42]
Loading Training Cameras [07/03 12:59:42]
Loaded Train Cameras: 711 [07/03 13:00:15]
Loaded Test Cameras: 0 [07/03 13:00:15]
Number of points at initialisation : 65890 [07/03 13:00:16]
Init density max: tensor(1.7833, device='cuda:0') [07/03 13:00:16]
torch.cuda.memory_allocated: 0.027213GB [07/03 13:00:17]
torch.cuda.memory_reserved: 0.074219GB [07/03 13:00:17]
torch.cuda.max_memory_reserved: 0.074219GB [07/03 13:00:17]
Training progress: 100%|██████████| 30000/30000 [4:31:29<00:00, 1.84it/s, Loss=0.0326953 Num Prim: 1505756 I: 95.8265609741211]

[ITER 7000] Evaluating train: L1 0.042052949965000155 PSNR 23.632291412353517 [07/03 14:00:21]

[ITER 7000] Saving Gaussians [07/03 14:00:21]

[ITER 30000] Evaluating train: L1 0.022222084552049638 PSNR 28.31905632019043 [07/03 17:31:47]

[ITER 30000] Saving Gaussians [07/03 17:31:47]

[half-potato]

28 PSNR doesn't seem bad. What does the result look like? Ignore the slang warning. Also, using 360 images projected to a perspective image will probably cause issues. Just use the 360 images raw and code a special lens to shoot the rays corresponding to the 360 camera.

[AntonyEddard]

The output is looking very cloudy and smoothed out. Maybe i will try with normal video and see how it works and let you know the results... also i will try with the dataset that you have given in github..

[yuancaimaiyi]

@half-potato
My dataset:
Because my machine has limited video memory of only 8G, 2600 images of 640480 images, in order to test ever, I reduced the resolution of 640480 by 4 times, that is, 160*120. The following is the effect of the original gaussian splatting and the effect of ever training:
It shows that I used the web https://antimatter15.com/splat/
Gaussian

splatting:

ever training

In addition, the PSNR of ever training is much larger than that of gaussian splatting, but the effect is very poor!

[half-potato]

An EVER "splat" is not the same as a gaussian "splat". It cannot be viewed in a splat viewer. It is likely much higher quality when viewed properly.

[yuancaimaiyi]

@half-potato
Thanks for the explanation. I thought the output of ever was the same as the output of gaussian splatting, but I encountered various problems when compiling SIBR_viewers. I almost gave up in the past few days. So I want to know if ever can be converted to gaussian splatting format so that I can use viser web to display it.

[coder12333444]

@half-potato
Hello, I have the same problem. I cannot compile the SIBR_viewers provided by your github, so I use gaussian splatting (ubuntu20.04). At the same time, I run docker_visualize.py and SIBR_remoteGaussian_app, but SIBR does not display anything. I read yuancaimaiyi's reply and I have the same problem. I want to convert the results of ever_training into the ordinary gaussian splatting format, so that we can use the popular web tools for visualization, which can also better promote your work. Otherwise, this obstacle will limit many people who want to learn ever.

[half-potato]

It is not possible to convert to a splat. I think the reason SIBR does not display anything is likely because it is not managing to connect. Did you run it on the same computer? Also, on what IP and PORT?

[coder12333444]

@half-potato
OK,I'm running on the same machine. I run
bash docker_visualize.py /gs_data/indoor_ever/model /gs_data/indoor_ever/train 6006 127.0.0.1
logs:Loading trained model at iteration 30000 [27/03 05:30:58]
docker ps is as follows:
f25f8f09664f halfpotato/ever:latest "/opt/nvidia/nvidia_…" 3 minutes ago Up 3 minutes 127.0.0.1:6006->6006/tcp, 6009/tcp quirky_bell
and the I run
./SIBR_remoteGaussian_app --ip 127.0.0.1 --port 6009

Always trying to connect...

[half-potato]

Try

./SIBR_remoteGaussian_app --ip 127.0.0.1 --port 6006

Instead. You seem to be using the wrong port.

[coder12333444]

@half-potato
port 6006 Same result, I have already tried it, and [SIBR] -- INFOS --: Connection dropped
[SIBR] -- INFOS --: Trying to connect...
[SIBR] -- INFOS --: Connected! will keep looping

[half-potato]

What's the error on the rendering side?