rotate matix of the dataset

[ChongjianGE]

Thanks for the wonderful work.

I have just noticed from the folder https://huggingface.co/datasets/tiange/Cap3D/tree/main/RenderedImage_CamMatrix_zips that there is no rotation matrix information. I am wondering if there is any way to obtain such information.

Thanks

[tiangeluo]

Hi Chongjian,

Have you tried unzip those compressed file? Each inside file should store a 3x4 extrinsic matrix (often referred to as a rotation-translation matrix, RT) for a camera in Blender. The RT matrix contains the 3x3 rotation matrix (R) and the 3x1 translation vector (T) in a 3x4 matrix format: RT = [R T].

Please let me know if you have further questions here.

[jiaosiyu1999]

Hi, I noticed that each .json file contains the following information. How do I extract the extrinsic matrix (RT) from it? Thanks.

{
    "max_depth": 5.0,
    "bbox": [
        [
            -0.2501941,
            -0.302039981,
            -0.450000018
        ],
        [
            0.2501941,
            0.302039981,
            0.450000018
        ]
    ],
    "origin": [
        1.37687612,
        0.368932843,
        0.497574925
    ],
    "x_fov": 0.691150367,
    "y_fov": 0.691150367,
    "x": [
        -0.258819163,
        0.965925753,
        -3.54346064E-08
    ],
    "y": [
        0.3183349,
        0.0852975845,
        -0.944133043
    ],
    "z": [
        -0.91196245,
        -0.244359732,
        -0.329564571
    ],
    "scale": [
        0.4499999,
        0.4499999,
        0.4499999
    ],
    "offset": [
        0.0,
        2.98023224E-08,
        0.0
    ]
}

[tiangeluo]

Hi @jiaosiyu1999 ,

Could you please try the below codes?

import json
import numpy as np

with open("xxx.json", "r") as f:
    data = json.load(f)

x_vector = data['x']
y_vector = data['y']
z_vector = data['z']
origin = data['origin']

rotation_matrix = np.array([x_vector, y_vector, z_vector]).T

translation_vector = np.array(origin)

rt_matrix = np.eye(4)
rt_matrix[:3, :3] = rotation_matrix
rt_matrix[:3, 3] = translation_vector

print("RT Matrix:")
print(rt_matrix)

The above results shoule be the same as the below:

data = json.load(open('transforms_train.json','r'))
# for frame 00000.png
data['frames'][0]['transform_matrix']

Btw, every 3D object is normalized via normalize_scene() detailed in https://huggingface.co/datasets/tiange/Cap3D/blob/main/RenderedImage_perobj_zips/README.md. If you are curious how exactly the json file being saved, please refer to dataset_readme.

Please let me know if you still have issues, I am here to assistant you on this case!

[jiaosiyu1999]

Thank you for your detailed reply. I have completed the 3D reconstruction by combining your code with Gaussian splatting.
The code for obtaining the R and T matrices is as follows. i.e., the rt_matrix requires additional np.linalg.inv transformation (although I'm not sure why this step is needed).

x_vector = data['x']
y_vector = data['y']
z_vector = data['z']
origin = data['origin']

rotation_matrix = np.array([x_vector, y_vector, z_vector]).T
translation_vector = np.array(origin)

rt_matrix = np.eye(4)
rt_matrix[:3, :3] = rotation_matrix
rt_matrix[:3, 3] = translation_vector

w2c = np.linalg.inv(rt_matrix)  # add this step !!!
R = np.transpose(w2c[:3,:3])  
T = w2c[:3, 3]

[tiangeluo]

Sounds good, and thank you for your quick attempts.

Since rt_matrix transforms coordinates from a reference frame to the new frame, its inverse as shown in your code, w2c, will transform coordinates from the new frame back to the reference frame.

[pkachana3]

Hi, quick follow up question -- how do you extract the camera intrinsic matrix (focal lengths, principal point)? I am trying to reconstruct the object by unprojecting depth using the intrinsics and extrinsics. Thanks!

[tiangeluo]

Hi @pkachana3 ,

Please refer to https://huggingface.co/datasets/tiange/Cap3D/blob/main/RenderedImage_perobj_zips/README.md, which provide related details. There is also an example file you could play with.

[msfdana110]

Hi Tiangeluo,
I am wondering how the latent space code (.pt files) for shapE are generated? I have been following the steps from .glb to .ply to get the point cloud and I know inside the pt file are 1024*1024 arrays, but not sure how these are generated. Can you please refer me to the part related to generating these latent space files. I know you fine tunning is based on the prompts and these latents files. Thanks a lot for your help.

[tiangeluo]

Hi @msfdana110 ,

Glad to help! The latent code is extracted using ShapE’s pre-trained encoder. You can find the code here: https://github.com/tiangeluo/DiffuRank/blob/main/shap-e/extract_latent.py