Cloud Log Service
- Release Notes and Announcements
- Announcements
- Product Introduction
- Limits
- Log Collection
- Purchase Guide
- Pay-as-You-Go
- Getting Started
- Operation Guide
- Resource Management
- Permission Management
- Log Collection
- Collection by LogListener
- LogListener Installation and Deployment
- Collecting Text Log
- Collecting Logs in Self-Built Kubernetes Cluster
- Importing Data
- Metric Collection
- Log Storage
- Search and Analysis (Log Topic)
- Statistical Analysis (SQL)
- SQL Syntax
- SQL Functions
- Search and Analysis (Metric Topic)
- Dashboard
- Creating Statistical Charts
- Data Processing documents
- Data Processing
- Data Processing Functions
- Scheduled SQL Analysis
- Shipping and Consumption
- Shipping to COS
- Shipping to CKafka
- Shipping to ES
- Log Shipping
- Consumption over Kafka
- Monitoring Alarm
- Managing Alarm Policies
- Channels to Receive Alarm Notifications
- Cloud Insight
- Historical Documentation
- Practical Tutorial
- Search and Analysis
- Monitoring Alarm
- Shipping and Consumption
- Developer Guide
- API Documentation
- Making API Requests
- Topic Management APIs
- Log Set Management APIs
- Topic Partition APIs
- Machine Group APIs
- Collection Configuration APIs
- Log APIs
- Metric APIs
- Alarm Policy APIs
- Data Processing APIs
- Kafka Protocol Consumption APIs
- CKafka Shipping Task APIs
- Kafka Data Subscription APIs
- COS Shipping Task APIs
- SCF Delivery Task APIs
- Scheduled SQL Analysis APIs
- COS Data Import Task APIs
- FAQs
- Health Check
- Collection
- Log Search
JSON Shipping
Last updated: 2024-01-20 17:44:35
Overview
You can log in to the CLS console and ship data in JSON format to COS. This document describes how to create a JSON shipping task.
Prerequisite
1. You have activated CLS, created a logset and a log topic, and successfully collected the log data.
2. You have activated COS and created a bucket in the target region for log topic shipping. For more information, see Creating Bucket.
3. Sub-accounts and collaborators need to be authorized by the root account. For more information on granting permissions, see CAM Access Management. For more information on copying authorization policies, see Examples of Custom Access Policies.
4. You have authorized the CLS service role to access COS. If you perform operations in the console, the system will guide you through the authorization process. If you directly call APIs, manual authorization will be required. For more information, see Viewing and Configuring Shipping Permissions.
Directions
1. Log in to the CLS console.
2. Click Log Topic on the left sidebar.
3. Click the desired log topic ID/name to go to the log topic management page.
4. Select the Ship to COS tab, click Add Shipping Configuration, and finish the configuration as detailed below.
The parameters are described as follows:
Configuration Item | Description | Limit | Required |
Shipping Task Name | Configures the name of a shipping task | The name can contain letters, numbers, underscores (_), and hyphens (-) | Yes |
COS Bucket | Target bucket for shipping. The target bucket must be in the same region as the current log topic. | A value selected from the drop-down list | Yes |
COS Path | COS bucket path, which is in the format of `/year/month/day/hour/` by default, such as `/2022/7/31/14/`. This format is used in COS for storing shipped log files. Here, the strftime syntax is supported. For example, if a log was shipped at 14:00 on July 31, 2022, the generated `/%Y/%m/%d/` path is `/2022/7/31/`, and the `/%Y%M%d/%H/` path is `/2022/07/31/14/`. | Cannot start with / | No |
Filename | Option 1 (recommended): Use the shipping time as the name. For example, `202208251645_000_132612782.gz` indicates the shipping time_log topic partition_offset. This type of files can be loaded in Hive.Option 2: Use a random number as the name. This is the legacy practice of naming files and cannot be recognized by Hive as it cannot recognize filenames starting with "_". You can add a custom prefix to the COS path, such as `/%Y%M%d/%H/Yourname`. | / | Yes |
Compression Format | To reduce read traffic fees, log files are compressed before being shipped to COS. Snappy, lzop, and GZIP are supported. | GZIP, Snappy, and lzop | Yes |
File Size | It indicates the size of the raw log file to be shipped and is used together with the shipping interval parameter. When either condition is met, compression will be performed accordingly. For example, if the size is set to 256 MB and the interval is set to 15 minutes, when the file reaches 256 MB within five minutes, the size condition will be met first to trigger shipping. | The value must be a number ranging from 5 to 256 in MB. | Yes |
Shipping Interval | It indicates the interval for shipping and is used together with the file size parameter. When either condition is met, compression will be performed accordingly. For example, if the size is set to 256 MB and the interval is set to 15 minutes, when the file reaches only 200 MB in 15 minutes, the shipping interval condition will be met first to trigger shipping. | Value range: 300-900s | Yes |
5. ClickNext to enter the Advanced Configuration page. Set Shipping Format to JSON and select the field to be shipped. Here,
_CONTENT_ is the user log data, and _SOURCE_, _FILENAME_, _HOSTNAME_, _TIMESTAMP_, and _TAG_ are CLS metadata fields that can be selected as needed.Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No