tencent cloud

Feedback

CSV Shipping

Last updated: 2024-01-20 17:44:35

    Overview

    You can log in to the CLS console and ship data in CSV format to COS. This document describes how to create a CSV shipping task.

    Prerequisite

    1. You have activated CLS, created a logset and a log topic, and successfully collected the log data.
    2. You have activated COS and created a bucket in the target region for log topic shipping. For more information, see Creating Bucket.
    3. Sub-accounts and collaborators need to be authorized by the root account. For more information on granting permissions, see CAM Access Management. For more information on copying authorization policies, see Examples of Custom Access Policies.
    4. You have authorized the CLS service role to access COS. If you perform operations in the console, the system will guide you through the authorization process. If you directly call APIs, manual authorization will be required. For more information, see Viewing and Configuring Shipping Permissions.

    Directions

    1. Log in to the CLS console.
    2. Click Log Topic on the left sidebar.
    3. Click the desired log topic ID/name to go to the log topic management page.
    4. Select the Ship to COS tab and click Add Shipping Configuration.
    5. Set the following configuration items.
    The configuration items are as described below:
    Configuration Item
    Description
    Limit
    Required
    Shipping Task Name
    Configures the name of a shipping task
    The name can contain letters, numbers, underscores (_), and hyphens (-)
    Yes
    COS Bucket
    Target bucket for shipping. The target bucket must be in the same region as the current log topic.
    A value selected from the drop-down list
    Yes
    COS Path
    COS bucket path, which is in the format of `/year/month/day/hour/` by default, such as `/2022/7/31/14/`. This format is used in COS for storing shipped log files. Here, the strftime syntax is supported. For example, if a log was shipped at 14:00 on July 31, 2022, the generated `/%Y/%m/%d/` path is `/2022/7/31/`, and the `/%Y%M%d/%H/` path is `/2022/07/31/14/`.
    Cannot start with /
    No
    Filename
    Option 1 (recommended): Use the shipping time as the name. For example, `202208251645_000_132612782.gz` indicates the shipping time_log topic partition_offset. This type of files can be loaded in Hive.Option 2: Use a random number as the name. This is the legacy practice of naming files and cannot be recognized by Hive as it cannot recognize filenames starting with "_". You can add a custom prefix to the COS path, such as `/%Y%M%d/%H/Yourname`.
    /
    Yes
    Compression Format
    To reduce read traffic fees, log files are compressed before being shipped to COS. Snappy, lzop, and GZIP are supported.
    GZIP, Snappy, and lzop
    Yes
    File Size
    It indicates the size of the raw log file to be shipped and is used together with the shipping interval parameter. When either condition is met, compression will be performed accordingly. For example, if the size is set to 256 MB and the interval is set to 15 minutes, when the file reaches 256 MB within five minutes, the size condition will be met first to trigger shipping.
    The value must be a number ranging from 5 to 256 in MB.
    Yes
    Shipping Interval
    It indicates the interval for shipping and is used together with the file size parameter. When either condition is met, compression will be performed accordingly. For example, if the size is set to 256 MB and the interval is set to 15 minutes, when the file reaches only 200 MB in 15 minutes, the shipping interval condition will be met first to trigger shipping.
    Value range: 300-900s
    Yes
    6. Click Next to enter the Advanced Configuration page. Set Shipping Format to CSV and set the configuration items.
    The configuration items are as described below:
    Configuration Item
    Description
    Limit
    Required
    Key
    Specifies the key name of the written CSV file. The value must be the key name or reserved parameter after logs are structured. Otherwise, the value is invalid.
    The name can contain letters, numbers, underscores (_), and hyphens (-)
    Yes
    Separator
    Separators between the fields in the CSV file
    A value selected from the drop-down list
    Yes
    Escape Character
    If the normal fields contain the selected separator characters, the separators will be enclosed by escape characters to prevent incorrect identification during data reading.
    A value selected from the drop-down list
    Yes
    Invalid Field Filling
    If the configured key field does not exist, an invalid field is filled in.
    A value selected from the drop-down list
    Yes
    Key in First Line
    Adds field name description to the first line of the CSV file. That is, the key value is written into the first line of the CSV file. This is disabled by default.
    Enabled/Disabled
    Yes
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support