tencent cloud

Feedback

Converting Audio to Expression

Last updated: 2023-02-27 12:20:01

    Overview

    This capability processes audio data and outputs blendshape data that meets the standards of Apple ARKit. For details, see ARFaceAnchor. You can pass the data to Unity to drive your model or use the data to implement other features.

    Integration

    Method 1: Integrate the Tencent Effect SDK

    The capability of converting audio to expressions is built into the Tencent Effect SDK, so to use the capability, you can integrate the Tencent Effect SDK.
    1. Download the complete edition of the Tencent Effect SDK.
    2. Follow the directions in Integrating Tencent Effect SDK to integrate the SDK.

    Method 2: Integrate the Audio-to-Expression SDK

    If you do not need other Tencent Effect capabilities, you can integrate the Audio-to-Expression SDK, the AAR file of which is about 6 MB. You can contact us to get the SDK.

    Directions

    1. Set the license. For detailed directions, see Integrating Tencent Effect SDK - Step 1. Authenticate.
    2. Configure the model files: Copy the model files from assets to a private directory of your application, such as context.getFilesDir() + "/my_models_dir/audio2exp". Then, call the init(String modelPath) API of Audio2ExpApi, passing in context.getFilesDir() + "/my_models_dir". You can find the model files at the following location of the SDK package:
    

    APIs

    API
    Description
    public int Audio2ExpApi.init(String modelPath);
    Initializes the SDK. You need to pass in the path of the model files. 0 indicates the initialization is successful.
    public float[] Audio2ExpApi.parseAudio(float[] inputData);
    The input is audio, which must be one-channel and have a sample rate of 16 Kbps and an array length of 267 (267 sampling points). The output is a float array with 52 elements, which correspond to 52 blendshapes. The value range of each element is 0-1, and their sequence is specified by Apple{"eyeBlinkLeft","eyeLookDownLeft","eyeLookInLeft","eyeLookOutLeft","eyeLookUpLeft","eyeSquintLeft","eyeWideLeft","eyeBlinkRight","eyeLookDownRight","eyeLookInRight","eyeLookOutRight","eyeLookUpRight","eyeSquintRight","eyeWideRight","jawForward","jawLeft","jawRight","jawOpen","mouthClose","mouthFunnel","mouthPucker","mouthRight","mouthLeft","mouthSmileLeft","mouthSmileRight","mouthFrownRight","mouthFrownLeft","mouthDimpleLeft","mouthDimpleRight","mouthStretchLeft","mouthStretchRight","mouthRollLower","mouthRollUpper","mouthShrugLower","mouthShrugUpper","mouthPressLeft","mouthPressRight","mouthLowerDownLeft","mouthLowerDownRight","mouthUpperUpLeft","mouthUpperUpRight","browDownLeft","browDownRight","browInnerUp","browOuterUpLeft","browOuterUpRight","cheekPuff","cheekSquintLeft","cheekSquintRight","noseSneerLeft","noseSneerRight","tongueOut"}
    public int Audio2ExpApi.release();
    Releases resources. Call this API when you no longer need the capability.

    Integration Code Sample

    @Override
    protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    findViewById(R.id.button).setOnClickListener(new OnClickListener() {
    @Override
    public void onClick(View view) {
    TELicenseCheck.getInstance().setTELicense(MainActivity.this, licenseUrl, licenseKey, new TELicenseCheckListener() {
    @Override
    public void onLicenseCheckFinish(int errorCode, String s) {
    Log.d(TAG, "onLicenseCheckFinish: errorCode = "+errorCode+",msg="+s);
    if (errorCode == TELicenseCheck.ERROR_OK) {
    //license check success
    Audio2ExpApi audio2ExpApi = new Audio2ExpApi();
    int err = audio2ExpApi.init(MainActivity.this.getFilesDir() +"/models");
    Log.d(TAG, "onLicenseCheckFinish: err="+err);
    //TODO start record and parse audio data
    } else {
    // license check failed
    }
    }
    });
    }
    });
    }
    Note
    For the full code sample, see Demos.
    For audio recording, see com.tencent.demo.avatar.audio.AudioCapturer.
    For more information on how to use the APIs, see com.tencent.demo.avatar.activity.Audio2ExpActivity and its related classes.
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support