Creating animation from videos

[timtensor]

Hello ,
Great work. I am trying to understand how , you managed managed to animate the character . If i understand correctly, there are two steps

a) 2d pose estimation using open pose or posenet
b) lifting the 2d pose to 3d . What did you use to achieve that ?
c) The correspodance of , 3d pose co -ordinates into untiy (x,y,z) co ordinates.

Is there a complete tutorial ? or Blog . I tried to follow your twitter account but could not understand many things. I am trying something similar to have animated avtars in unity for a more interactive teaching for school . Any help would be appreicated

[yukihiko]

I'm sorry. I did not notice at all. It seems that he overlooked the email.

I'll answer the question.
a) I am using a model that I learned with pytorch. This model is based on MobileNet.
The concept of HeatMap was based on PoseNet.

b) Created Dataset with 3D characters using Unity.

c) Please refer to the code of ThreeDPoseScript.cs.

Best regards.

[peterchen2001]

Hi yukihiko,
I am trying to understand the code, I got below questions while analyzing your sample demo:
1, What do these 369,373,etc numbers stand for? any references?
Onnx.Forward(outputs, new string[] { "369", "373", "361", "365" });
2,why the HeatMapCol's size is set to 14?
private const int HeatMapCol = 14;
private const int HeatMapCol_Squared = 14 * 14;
private const int HeatMapCol_Cube = 14 * 14 * 14;

[yukihiko]

1. Name of the output layer of the onnx file. Because it is an automatically assigned name, it is a number.
2. I am learning to output a 14x14x14 heat map during learning. The reason for 14 was just easy to calculate by convolution.

[peterchen2001]

Thanks for quick reply!