About the training process

[guoguoguilai]

Hello,
thank you so much for your amazing work ! 👍
I was wondering about a couple of things :

1. I am really curious about the architecture of the network. In other GitHub issues, you said that 2D and 3D heatmap are independent and trained at the same time. How to feed 3D data into the network? Does 2D heatmap and offset were used in predict stage?
2. How did you generate the coordinates of 3D data through unity. For the data collection stage , how does the action in front of the camera correspond to the action of avatar in the unity?
3. What is the meaning of 2D and 3D offset？

I'll really appreciate your help.
Thank you

[HiroDigi]

Hi,
I am hiro, yukihiro is my boss.
Nowaday he is busy so that I'll answer on behalf of him.

the architecture is based on Mobile net (or ResNet is also available) and
the output layer adobts PoseNet.
And the machine have learned 3D models which he had created by himself by Unity
and 2D image which is open to public.
The size of the 2D images are many different so that those size and coordinates has been arranged for the target image size.

In addition, the 2D images has not been used in predict stage in this sample.
Mainly those of images were used to over come the image domain,
then he has only CG model created by Unity for learning 3D model, and it tends to make the machine strong just for CG.

Firstly he gets the z-coordinate of the avatar,
and then arrange the camera to catch the view of specified width and height from the point.

Offset is vectors which indicate more precise position of target from each of points in heatmap.
By utilizing this method which uses heat map and offset vectors, the accuracy becomes better.
You can see the detail of this in this URL(https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5)

Thanks