tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Tencent Effect SDK
    Documentation
    Download PDF

    iOS

    Last updated: 2023-02-27 12:15:21
    Download PDF

      Overview

      
      This capability processes audio data and outputs blendshape data that meets the standards of Apple ARKit. For details, see ARFaceAnchor. You can pass the data to Unity to drive your model or use the data to implement other features.

      Integration

      Method 1: Integrate the Tencent Effect SDK

      1. The capability of converting audio to expressions is built into the Tencent Effect SDK, so to use the capability, you can integrate the Tencent Effect SDK.
      2. Download the complete edition of the Tencent Effect SDK.
      3. Follow the directions in Integrating Tencent Effect SDK to integrate the SDK.
      4. Import Audio2Exp.framework in the SDK to your project. Select your target, under the General tab, find Frameworks,Libraries, and Embedded Content, and set Audio2Exp.framework to Embed & Sign.

      Method 2: Integrate the Audio-to-Expression SDK

      If you only need the capability of converting audio to expressions, you can integrate the Audio-to-Expression SDK (Audio2Exp.framework is about 7 MB). Import the two dynamic frameworks Audio2Exp.framework and YTCommonXMagic.framework to your project. Select your target, under the General tab, find Frameworks,Libraries, and Embedded Content, and set Audio2Exp.framework and YTCommonXMagic.framework to Embed & Sign.

      Directions

      1. Set the license. For detailed directions, see Integrating Tencent Effect SDK - Step 1. Authenticate.
      2. Configure the model file. Copy the model file audio2exp.bundle to your project directory. When calling initWithModelPath: of Audio2ExpApi, pass in the path of model file.

      APIs

      API
      Description
      + (int)initWithModelPath:(NSString*)modelPath;
      Initializes the SDK. You need to pass in the path of the model file. 0 indicates the initialization is successful.
      + (NSArray )parseAudio:(NSArray )inputData;
      The input is audio, which must be one-channel and have a sample rate of 16 Kbps and an array length of 267 (267 sampling points). The output is a float array with 52 elements, which correspond to 52 blendshapes. The value range of each element is 0-1, and their sequence is specified by Apple{"eyeBlinkLeft","eyeLookDownLeft","eyeLookInLeft","eyeLookOutLeft","eyeLookUpLeft","eyeSquintLeft","eyeWideLeft","eyeBlinkRight","eyeLookDownRight","eyeLookInRight","eyeLookOutRight","eyeLookUpRight","eyeSquintRight","eyeWideRight","jawForward","jawLeft","jawRight","jawOpen","mouthClose","mouthFunnel","mouthPucker","mouthRight","mouthLeft","mouthSmileLeft","mouthSmileRight","mouthFrownRight","mouthFrownLeft","mouthDimpleLeft","mouthDimpleRight","mouthStretchLeft","mouthStretchRight","mouthRollLower","mouthRollUpper","mouthShrugLower","mouthShrugUpper","mouthPressLeft","mouthPressRight","mouthLowerDownLeft","mouthLowerDownRight","mouthUpperUpLeft","mouthUpperUpRight","browDownLeft","browDownRight","browInnerUp","browOuterUpLeft","browOuterUpRight","cheekPuff","cheekSquintLeft","cheekSquintRight","noseSneerLeft","noseSneerRight","tongueOut"}
      + (int)releaseSdk
      Releases resources. Call this API when you no longer need the capability.

      Integration Code Sample

      // Initialize the Audio-to-Expression SDK
      NSString *path = [[NSBundle mainBundle] pathForResource:@"audio2exp" ofType:@"bundle"];
      int ret = [Audio2ExpApi initWithModelPath:path];
      // Convert audio to blendshape data
      NSArray *emotionArray = [Audio2ExpApi parseAudio:floatArr];
      // Release the SDK
      [Audio2ExpApi releaseSdk];
      
      // Use with the Tencent Effect SDK
      // Initialize the SDK
      self.beautyKit = [[XMagic alloc] initWithRenderSize:previewSize assetsDict:assetsDict];
      // Load the avatar materials
      [self.beautyKit loadAvatar:bundlePath exportedAvatar:nil completion:nil];
      // Pass the blendshape data to the SDK, and the effects will be applied.
      [self.beautyKit updateAvatarByExpression:emotionArray];
      Note:
      For the full code sample, see Demos.
      For audio recording, see TXCAudioRecorder.
      For more information on using the APIs, see VoiceViewController and the related classes.
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy