tencent cloud

Feedback

Last updated: 2023-05-18 16:33:49

    Overview

    This capability processes the OpenGL textures of the camera and outputs 3D body data. You can pass the data to Unity to drive your model or use the data to implement other features.

    Integration (Android)

    Follow the directions in Integrating Tencent Effect SDK to integrate the Tencent Effect SDK.

    API calls

    1. Enable the feature (XmagicApi.java).
    public void setFeatureEnableDisable(String featureName, boolean enable);
    Set featureName to XmagicConstant.FeatureName.BODY_3D_POINT.
    2. Configure the data callback (XmagicApi.java).
    Version 2.6.0 and earlier versions use the following method.
    void setYTDataListener(XmagicApi.XmagicYTDataListener ytDataListener)
    
    public interface XmagicYTDataListener {
    void onYTDataUpdate(String data)
    }
    onYTDataUpdate returns a JSON string. You can find an example below.
    face_info is facial data, which is not used by this capability.
    For the meanings of fields in body_3d_info, see below.
    
    Version 3.0.0 uses the following method.
    void setAIDataListener(XmagicApi.OnAIDataListener aiDataListener)
    
    public interface OnAIDataListener {
    void onFaceDataUpdated(List<TEFaceData> faceDataList);
    void onHandDataUpdated(List<TEHandData> handDataList);
    void onBodyDataUpdated(List<TEBodyData> bodyDataList);
    void onAIDataUpdated(String data); //This method is a new method added in version 3.0.0, and the data structure is consistent with the XmagicYTDataListener interface used in previous versions.
    }
    onAIDataUpdated returns a JSON string. You can find an example below.
    face_info is facial data, which is not used by this capability.
    For the meanings of fields in body_3d_info, see below.

    Body Keypoints and Descriptions

    SMPL keypoints
    
    SMPL-X hand keypoints
    
    
    Here is an example of the JSON string output by the SDK:
    
    The following section details the fields in body_3d_info:
    imageWidth and imageHeight: The width and height of images sent to the SDK.
    items: An array. Currently, it contains only one element.
    index: A reserved field. You can ignore it.
    pose:
    1. Location [0,2]. The 3D location (xyz) of the body root bone with the camera at the center.
    2. Location [3,12]. The body shape. It includes 10 floating numbers, which are based on 10 meshes of SMPL.
    3. Location [13]. The focal length, which is 5000.
    4. Location [14,29]. The OpenGL projection matrix in a 3D space, which is generated based on the focal length. A 4 x 4 projection matrix is calculated as follows in the algorithm:
    matrix={
    2 * focal_length / img_wid, 0, 0, 0,
    0, 2 * focal_length / img_hei, 0,0,
    0,0, (zf + zn) / (zn - zf), -1,
    0, 0, (2.0f * zf * zn) / (zn - zf), 0};
    }
    5. Location [30,33]. Whether the left toe, left heel, right toe, and right heel are on the ground.
    position_x,position_y,position_z:
    1. Location [0,23]. 2D body keypoints (figure 1 above). The value of position_z for all 2D keypoints is 0.
    2. Location [24,47]. 3D body keypoints (figure 1 above).
    rotation
    1. Location [0,23]. The body bone rotation quaternions (wxyz).
    2. Location [25,54]. The hand bone rotation quaternions (wxyz). There are 15 quaternions for each hand.

    Bone Names

    No.
    Bone Name
    Bone Name 2
    0
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    "pelvis",
    "left_hip",
    "right_hip",
    "spine1",
    "left_knee",
    "right_knee",
    "spine2",
    "left_ankle",
    "right_ankle",
    "spine3",
    "left_foot",
    "right_foot",
    "neck",
    "left_collar",
    "right_collar",
    "head",
    "left_shoulder",
    "right_shoulder",
    "left_elbow",
    "right_elbow",
    "left_wrist",
    "right_wrist",
    "left_hand"
    "right_hand"
    "Hips"
    "LeftUpLeg"
    "RightUpLeg"
    "Spine"
    "LeftLeg"
    "RightLeg"
    "Spine1"
    "LeftFoot"
    "RightFoot"
    "Spine2"
    ""
    ""
    "Neck"
    "LeftShoulder"
    "RightShoulder"
    "Head"
    "LeftArm"
    "RightArm"
    "LeftForeArm"
    "RightForeArm"
    "LeftHand"
    "RightHand"
    ""
    ""
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    "left_index1"
    "left_index2"
    "left_index3"
    "left_middle1"
    "left_middle2"
    "left_middle3"
    "left_pinky1"
    "left_pinky2"
    "left_pinky3"
    "left_ring1"
    "left_ring2"
    "left_ring3"
    "left_thumb1"
    "left_thumb2"
    "left_thumb3
    IndexFinger1_L
    IndexFinger2_L
    IndexFinger3_L
    MiddleFinger1_L
    MiddleFinger2_L
    MiddleFinger3_L
    PinkyFinger1_L
    PinkyFinger2_L
    PinkyFinger3_L
    RingFinger1_L
    RingFinger2_L
    RingFinger3_L
    ThumbFinger1_L
    ThumbFinger2_L
    ThumbFinger3_L
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    "right_index1"
    "right_index2"
    "right_index3"
    "right_middle1"
    "right_middle2"
    "right_middle3"
    "right_pinky1"
    "right_pinky2"
    "right_pinky3"
    "right_ring1"
    "right_ring2"
    "right_ring3"
    "right_thumb1"
    "right_thumb2"
    "right_thumb3"
    IndexFinger1_R
    IndexFinger2_R
    IndexFinger3_R
    MiddleFinger1_R
    MiddleFinger2_R
    MiddleFinger3_R
    PinkyFinger1_R
    PinkyFinger2_R
    PinkyFinger3_R
    RingFinger1_R
    RingFinger2_R
    RingFinger3_R
    ThumbFinger1_R
    ThumbFinger2_R
    ThumbFinger3_R
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support