Statistical Metrics

You can click Real-time Operation and Maintenance > Operations﻿ More ﻿ Statistical Metrics or Real-time Operation and Maintenance ﻿ Task Name ﻿ Statistical Metrics to enter the Real-time Synchronization Statistical Metrics interface to view task synchronization progress, task resource occupancy, and database table synchronization status. The statistical metrics are mainly divided into three dimensions: Read and Write Metrics, Resource Metrics, and Link Details (for whole database tasks only).
Read and Write Metrics
The Read and Write Metrics page allows you to view the Full Incremental Synchronization Progress of real-time synchronization tasks by Data Source dimension, the number of task data synchronizations by Time Dimension, as well as key metrics of the overall task such as read-write speed, throughput, and latency.
Task Full Incremental Sync Progress
The Full Incremental Synchronization Progress of tasks can be viewed by Data Source dimension across three stages: pre-task environment preparation, Full Synchronization, and Incremental Synchronization.
Note:
Currently, viewing the Full Incremental Synchronization Progress of tasks is supported only when the source is MySQL, Oracle, and PostgreSQL.
Synchronization Stage
Stage Overview
Display Metrics
Task Pre-environment Preparation
This stage mainly initiates the corresponding task
Display specific progress (percentage) and status (Not Started/In Progress/Completed)
Full Synchronization Phase
Read all existing data from the source database table
In this stage, only the number of tables with successfully read and existing data are counted and displayed, showing the start time, end time, and specific progress (percentage) and status (Not Started/In Progress/Completed) of this stage
Incremental Synchronization Phase
 Consume change data from binlog and read incremental data from the source database table
In this stage, only the number of tables with successfully read and new data are counted and displayed, showing the start time, end time, and synchronization status (Not Started/In Progress/Completed) of this stage
When the task reaches the incremental phase, it will normally remain in the  in-progress  state without a specific end time.
Meanwhile, the task business latency indicator is displayed on the right side of the status, refreshed automatically every 5 seconds.
Note:
The incremental phase may exhibit some business latency, which is normal. However, if the latency is significant, please monitor its impact on the business.
Synchronization Overview
The synchronization overview allows you to view the total data synchronization of real-time tasks according to different time dimensions and intervals:
Parameters
Description
Time period
You can select the last hour, today, this week, from the start of operation, or from a custom time range
Total Reads
The actual number of records read from the source during the execution of this task. This metric does not include the total amount of data eliminated through filtering or other methods.
Total Successful Writes
The total amount of data successfully written to the target during the execution of this task.
Total Dirty Data
The total amount of data that failed to be written to the target during the execution of this task. This metric does not include data that was not written due to active ignore/filter settings in the task configuration. This includes specific stops, abnormal restarts, other running policies, and data filtering.
Total Running Time
The total running time accumulated since the task started, including any pause time.
Read and Write Details
Read and Write Details enable you to view the read and write speed of real-time tasks and latency-related metrics by different time dimensions and intervals.
Parameters
Description
Time period
You can select the last hour, today, this week, from the start of operation, or from a custom time range
Read Speed
Total number of reads within the statistical interval / statistical interval
Read Throughput 
Total read amount within the statistical interval / statistical interval
Write Speed
Number of successful writes within the statistical interval / statistical interval
Write Throughput
 Total successful writes within the statistical interval / statistical interval
Write Latency
Link latency from Source end to Sink end, where write latency = System Time - Record Read Time (LatencyMarker timestamp on the reading end)
Number of Job Restarts
Current number of job restarts
Log position information
Real-time task CDC log position information
Job Transmission Latency
Time when data is read by Source (FetchTime) - Data event time (EventTime).
Job Business Latency
Time when data leaves Source (EmitTime) - Data event time (EventTime)
Backpressure Metrics
This metric reflects the relative speed of the reading end and writing end. Below 0.5% indicates the reading end is slower than the writing end; above 0.5% indicates the reading end is faster than the writing end
Resource Metrics
CheckPoint
Metric Parameters
Description
Sample value
Last Checkpoint Size
The size of the most recent checkpoint for the current job
751321.00Bytes
Checkpoint Duration
The time taken for the current job's checkpoint
723.00ms
Checkpoint Failure Count
The total number of checkpoint failures for the current job
8 times
TaskManager
Metric Parameters
Description
Sample value
TaskManager CPU utilization
The CPU utilization of the TaskManager for the current job.
7.12%
The amount of heap memory used by the TaskManager.
The amount of heap memory used by the TaskManager for the current job.
1040001560.00Bytes
TaskManager Old Generation GC Count
The number of old generation GCs in the TaskManager for the current job.
3 times
TaskManager Old Generation GC Time
The total time spent on old generation GCs in the TaskManager for the current job.
701.00ms
The total physical memory usage (RSS) of the TaskManager.
The total physical memory usage (RSS) of the JVM for the TaskManager in the current job, including heap, off-heap, native, and other memory areas.
3597035110.00Bytes
JobManager
Metric Parameters
Description
Sample value
JM CPU Load
The recent CPU utilization of the JVM for the TaskManager.
12%
JM Head Memory
The current heap memory usage of the TaskManager.
1 time
JM GC Count
Status.JVM.GarbageCollector at the TaskManager level.<GarbageCollector>.Count, GC (garbage collection) count.
5 times
JM GC Time
Status.JVM.GarbageCollector at the TaskManager level.<GarbageCollector>.Time, GC (garbage collection) time.
65ms
Link Details
Maintenance support for whole database synchronization tasks includes viewing synchronization link details. Click  Real-time Operation and Maintenance >  Task Name > Statistics Metrics > Link Details  to access the link details maintenance page. Link details allow you to view data read/write metrics at the table granularity.
Read End
Source Table Metrics: Shows detailed statistics of tables successfully read in whole database synchronization tasks.
Including database name, table name, read rows/bytes, read speed/throughput.
Trend charts support viewing detailed metrics of single tables during task execution.
Search by table name or database name is supported.
Trend Chart
By default, the trend chart of various metrics in the past hour is expanded, and the display period can be adjusted manually.
Data in the pop-up window is automatically refreshed every minute, and manual refresh is supported.
Write End
Target Table Metrics: Shows table statistics for whole database tasks.
Including database name, table name, write rows/bytes, write speed/throughput, dirty data.
Trend charts support viewing detailed metrics of single tables during task execution.
Search by table name or database name is supported.
Trend Chart
By default, the trend chart of various metrics in the past hour is expanded, and the display period can be adjusted manually.
Data in the pop-up window is automatically refreshed every minute, and manual refresh is supported.

Was this page helpful?

You can also Contact Sales or Submit a Ticket for help.

Yes

Synchronization Stage	Stage Overview	Display Metrics
Task Pre-environment Preparation	This stage mainly initiates the corresponding task	Display specific progress (percentage) and status (Not Started/In Progress/Completed)
Full Synchronization Phase	Read all existing data from the source database table	In this stage, only the number of tables with successfully read and existing data are counted and displayed, showing the start time, end time, and specific progress (percentage) and status (Not Started/In Progress/Completed) of this stage
Incremental Synchronization Phase	Consume change data from binlog and read incremental data from the source database table	In this stage, only the number of tables with successfully read and new data are counted and displayed, showing the start time, end time, and synchronization status (Not Started/In Progress/Completed) of this stage When the task reaches the incremental phase, it will normally remain in the in-progress state without a specific end time. Meanwhile, the task business latency indicator is displayed on the right side of the status, refreshed automatically every 5 seconds. Note: The incremental phase may exhibit some business latency, which is normal. However, if the latency is significant, please monitor its impact on the business.

Parameters	Description
Time period	You can select the last hour, today, this week, from the start of operation, or from a custom time range
Total Reads	The actual number of records read from the source during the execution of this task. This metric does not include the total amount of data eliminated through filtering or other methods.
Total Successful Writes	The total amount of data successfully written to the target during the execution of this task.
Total Dirty Data	The total amount of data that failed to be written to the target during the execution of this task. This metric does not include data that was not written due to active ignore/filter settings in the task configuration. This includes specific stops, abnormal restarts, other running policies, and data filtering.
Total Running Time	The total running time accumulated since the task started, including any pause time.

Metric Parameters	Description	Sample value
Last Checkpoint Size	The size of the most recent checkpoint for the current job	751321.00Bytes
Checkpoint Duration	The time taken for the current job's checkpoint	723.00ms
Checkpoint Failure Count	The total number of checkpoint failures for the current job	8 times

tencent cloud

Sign Up

Log in

Compute

Microservice

Data Migration

Database SaaS Tool

Data Security

Application Security

Big Data

Voice Technology

Internet of Things

Stream Services

Cloud Real-time Rendering

Cloud Resource Management

More

Edge Computing

Serverless

Relational Database

Networking

Business Security

Domains & Websites

Face Recognition

AI Platform Service

Middleware

Media On-Demand

Game Services

Management and Audit Tools

Container

Essential Storage Service

Enterprise Distributed DBMS

CDN and Acceleration

Security Services

Enterprise Applications

Tencent Big Model

Natural Language Processing

Communication

Media Process Services

Education Sevices

Developer Tools

Distributed cloud

Data Process and Analysis

NoSQL Database

Network Security

Cloud Security

Office Collaboration

Image Creation

Optical Character Recognition

Interactive Video Services

Media SDK

Medical Services

Monitor and Operation

Read and Write Metrics

Task Full Incremental Sync Progress

Synchronization Overview

Read and Write Details

Resource Metrics

CheckPoint

TaskManager

JobManager

Link Details

Read End

Write End