Transfer PartNet latent code(size=part_num, 128) to SDM hidden code(size=1, 32)

[susan31213]

Hi, Thanks for your help in resolving the reproduce issue last time. Now I'm trying to replace some parts of model with another model.
I have two model's sdm latent codes whose size are both (1, 32) and they look like this:

My goal is replacing Model.1's chair back with Model.2's back.
First, I got their PartNet code by:

# 1st model's sdm code(1, 32) -> partnet codes "recoversym1"(8, 128)
latent1 = np.expand_dims(latent1, axis=0)
recover1 = sess.run([self.embedding_decode[self.part_num]], feed_dict = {self.embedding_inputs[self.part_num]: latent1})[0]
recoversym1 = np.reshape(recover1, (len(recover1), self.part_num, self.part_dim+self.hiddendim[0]))

# 2nd model's sdm code(1, 32) -> partnet codes "recoversym2"(8, 128)
latent2 = np.expand_dims(latent2, axis=0)
recover2 = sess.run([self.embedding_decode[self.part_num]], feed_dict = {self.embedding_inputs[self.part_num]: latent2})[0]
recoversym2 = np.reshape(recover2, (len(recover2), self.part_num, self.part_dim+self.hiddendim[0]))

Next, I replaced the back part of Model.1's PartNet code with the back part of Model.2's PartNet code, and generate the obj file.

# create a new object partnet code, copy 2nd model's chair back to 1st model
edit_part_idx = 2
recoversym3 = recoversym1.copy()
recoversym3[:, edit_part_idx, :] = recoversym2[:, edit_part_idx, :]
if self.change_net:                         # enter here
    symmetryf3 = np.concatenate([np.expand_dims(recover_datasymv2(recoversym3[:,k,-self.part_dim:], datainfo.boxmin[k], datainfo.boxmax[k]), axis = 1) for k in range(self.part_num)], axis = 1)
else:
    symmetryf3 = recover_datasym(recoversym3[:,:,-self.part_dim:], datainfo.boxmin[0], datainfo.boxmax[0])

# generate edited obj
recovers = None
for k in range(self.part_num):
    recover = sess.run([self.embedding_decode[k]], feed_dict = {self.embedding_inputs[k]: recoversym3[:,k,:self.hiddendim[0]]})[0]
    deforma_v1, deforma_v_align = sess.run([self.deform_vertex[k], self.deform_vertex_align[k]], feed_dict={self.feature2point: recover, self.controlpoint: np.tile([0,0,0], (np.shape(recover)[0], 1, 1))})
    deforma_v1 = self.alignallmodels(deforma_v1, id = k)
    deformas.append(deforma_v1)
    self.v2objfile(deforma_v1, path + '/gt_struc_' + self.part_name[k], ['0'], '0', self.part_name[k])
    if k == 0:
        recovers = recover.copy()
    else:
        recovers = np.concatenate([recovers, recover.copy()])

The obj file looks like this:

Looks great! 👍

But when I encode its shape feature whose size is [8, 3752, 9] into sdm latent code:

# encode shape features(8, 3752, 9) -> sdm latent code(1, 32)
if advance_api:
    app_handle = sess.run(self.app_iterator.string_handle())
for i in range(self.part_num+1):
    sess.run(self.app_iterator.initializer, feed_dict={self.inputs_feature: np.expand_dims(recovers, axis=0), self.inputs_symmetry: symmetryf3})
    while True:
        try:
            embedding = sess.run([self.encode[i]], feed_dict = {self.handle: app_handle})[0]
        except tf.errors.OutOfRangeError:
            break
    print(embedding.shape)  # (1, 32)

and decode the sdm code to obj file:

# decode sdm latent code to obj file, this should be like "gt_struc" obj
recover_from32 = sess.run([self.embedding_decode[self.part_num]], feed_dict = {self.embedding_inputs[self.part_num]: embedding})[0]
recoversym_from32 = np.reshape(recover_from32, (len(recover_from32), self.part_num, self.part_dim+self.hiddendim[0]))
for k in range(self.part_num):
    recover = sess.run([self.embedding_decode[k]], feed_dict = {self.embedding_inputs[k]: recoversym_from32[:,k,:self.hiddendim[0]]})[0]
    deforma_v1, deforma_v_align = sess.run([self.deform_vertex[k], self.deform_vertex_align[k]], feed_dict={self.feature2point: recover, self.controlpoint: np.tile([0,0,0], (np.shape(recover)[0], 1, 1))})
    deforma_v1 = self.alignallmodels(deforma_v1, id = k)
    deformas.append(deforma_v1)
    self.v2objfile(deforma_v1, path + '/pred_struc_' + self.part_name[k], ['0'], '0', self.part_name[k])

The obj file looks like this, and it doesn't looks like the result in above image: 😢

I'm wondering the problem is the order in the shape feature of preprocessing data and inferenced data are different, so I got a wrong sdm latent code 🤔 ?

Thanks for reading this issues and I will be grateful for any help you can provide.

Regards,
Susan

[susan31213]

Here is my test code and model sdm latent codes: https://1drv.ms/u/s!AkQAmtyxosCJwF24G_fT2ifvqR9R?e=ISxfOt
If you can run the command successfully in README.md, put the scripts and code files into code/python/chair_reproduce and run to reproduce my result 😃 :

python test_copy_paste.py --output_dir ./05060123_6863bin_1-joint_1-l0_100.0-l2_10.0-l3_1.0-l4_0.001-model_chair-trcet_1.0 --latent1 model1.npz --latent2 model2.npz

[susan31213]

Some update:
I tried a simpler case that encode and decode a sdm hidden code with below codes.
I also uploaded this code to the drive folder: https://1drv.ms/u/s!AkQAmtyxosCJwF24G_fT2ifvqR9R?e=ISxfOt

def encoder_and_decode(self, sess, datainfo, latent, path):
# model's sdm code(1, 32) -> partnet codes(8, 128)
latent = np.expand_dims(latent, axis=0)
recover = sess.run([self.embedding_decode[self.part_num]], feed_dict = {self.embedding_inputs[self.part_num]: latent})[0]
recoversym = np.reshape(recover, (len(recover), self.part_num, self.part_dim+self.hiddendim[0]))
symmetryf = np.concatenate([np.expand_dims(recover_datasymv2(recoversym[:,k,-self.part_dim:], datainfo.boxmin[k], datainfo.boxmax[k]), axis = 1) for k in range(self.part_num)], axis = 1)

# generate edited obj
recovers = None
for k in range(self.part_num):
    recover = sess.run([self.embedding_decode[k]], feed_dict = {self.embedding_inputs[k]: recoversym[:,k,:self.hiddendim[0]]})[0]
    deforma_v1, deforma_v_align = sess.run([self.deform_vertex[k], self.deform_vertex_align[k]], feed_dict={self.feature2point: recover, self.controlpoint: np.tile([0,0,0], (np.shape(recover)[0], 1, 1))})
    deforma_v1 = self.alignallmodels(deforma_v1, id = k)
    self.v2objfile(deforma_v1, path + '/gt_struc_' + self.part_name[k], ['0'], '0', self.part_name[k])
    if k == 0:
        recovers = recover.copy()
    else:
        recovers = np.concatenate([recovers, recover.copy()])


# encode shape features(8, 3752, 9) -> sdm latent code(1, 32)
if advance_api:
    app_handle = sess.run(self.app_iterator.string_handle())
for i in range(self.part_num+1):
    sess.run(self.app_iterator.initializer, feed_dict={self.inputs_feature: np.expand_dims(recovers, axis=0), self.inputs_symmetry: symmetryf})
    while True:
        try:
            embedding = sess.run([self.encode[i]], feed_dict = {self.handle: app_handle})[0]
        except tf.errors.OutOfRangeError:
            break
    print(embedding.shape)  # (1, 32)

# decode sdm latent code to obj file, this should be like "gt_struc" obj
recover_from32 = sess.run([self.embedding_decode[self.part_num]], feed_dict = {self.embedding_inputs[self.part_num]: embedding})[0]
recoversym_from32 = np.reshape(recover_from32, (len(recover_from32), self.part_num, self.part_dim+self.hiddendim[0]))
for k in range(self.part_num):
    recover = sess.run([self.embedding_decode[k]], feed_dict = {self.embedding_inputs[k]: recoversym_from32[:,k,:self.hiddendim[0]]})[0]
    deforma_v1, deforma_v_align = sess.run([self.deform_vertex[k], self.deform_vertex_align[k]], feed_dict={self.feature2point: recover, self.controlpoint: np.tile([0,0,0], (np.shape(recover)[0], 1, 1))})
    deforma_v1 = self.alignallmodels(deforma_v1, id = k)
    self.v2objfile(deforma_v1, path + '/pred_struc_' + self.part_name[k], ['0'], '0', self.part_name[k])

# save inter_sym.mat
sio.savemat(path+'/inter_sym_pred.mat', {'symmetry_feature':symmetryf, 'emb':embedding})
np.savez(f'{path}/embedding.npz', z=embedding)

return

Still, no matter what latent I input, I always got the same chair model(the last image in my first comment).
Is this a bug? or I got it wrong?

Thanks & Regards,
Susan

[susan31213]

Hello, @shinxg, sorry to bother you.
Do you have any idea about what caused this strange result?

[tommaoer]

Hi, @susan31213, I think you could check whether the latent code of the wrong model is the same as the previous model. If the latent code is the same, the results are also the same, right?
If it is changed, you must check who changes it. And you could also check the deformation gradients of the output shape and input shape to discover more details.

Thanks for your interest on our work!
If you have any further problems, feel free to contact me.

Best,
Jie

[susan31213]

Hello @jie,
Thanks for your advice, I checked the sdm latent code and they are very different... and I found I input a wrong inputs_symmetry. I need to input a raw symmetry feature but not a recovered feature with bbox_min and bbox_max.
Now I can encode a model information to a sdm latent code and decode it to the origin model correctly. Thank you very much!

But now I encounter another problem in the copy-paste scenario. I only edit the chair's back, and I re-encode the edited model to a sdm latent code and decode it, but got the result like this:

well... the chair's back looks "flatter" on its edge than original model , but doesn't look like the reconstruction from PartNet code. 🤔
I also tried to edit other parts of the model, here is a better result I got:

and from these results, I suppose that 32-dimension sdm latent code can not "fully describe" the reconstruction from 8 ×128-dim PartNet codes.
To conquer this problem, did you try to use a larger dimension for sdm latent code? If yes, Could you share the pre-trained model, or share the training data and the training guideline? because I can't find any information about training SDM-Net in this repo. 🥺

Best,
Susan