tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Tencent Cloud WeData
    Documentation
    Download PDF

    Statistical Metrics

    Last updated: 2024-11-01 17:19:59
    Download PDF
      You can click Real-time Operation and Maintenance > Operations More  Statistical Metrics or Real-time Operation and Maintenance  Task Name  Statistical Metrics to enter the Real-time Synchronization Statistical Metrics interface to view task synchronization progress, task resource occupancy, and database table synchronization status. The statistical metrics are mainly divided into three dimensions: Read and Write Metrics, Resource Metrics, and Link Details (for whole database tasks only).

      Read and Write Metrics

      The Read and Write Metrics page allows you to view the Full Incremental Synchronization Progress of real-time synchronization tasks by Data Source dimension, the number of task data synchronizations by Time Dimension, as well as key metrics of the overall task such as read-write speed, throughput, and latency.

      Task Full Incremental Sync Progress

      The Full Incremental Synchronization Progress of tasks can be viewed by Data Source dimension across three stages: pre-task environment preparation, Full Synchronization, and Incremental Synchronization.
      Note:
      Currently, viewing the Full Incremental Synchronization Progress of tasks is supported only when the source is MySQL, Oracle, and PostgreSQL.
      Synchronization Stage
      Stage Overview
      Display Metrics
      Task Pre-environment Preparation
      This stage mainly initiates the corresponding task
      Display specific progress (percentage) and status (Not Started/In Progress/Completed)
      Full Synchronization Phase
      Read all existing data from the source database table
      In this stage, only the number of tables with successfully read and existing data are counted and displayed, showing the start time, end time, and specific progress (percentage) and status (Not Started/In Progress/Completed) of this stage
      Incremental Synchronization Phase
      Consume change data from binlog and read incremental data from the source database table
      In this stage, only the number of tables with successfully read and new data are counted and displayed, showing the start time, end time, and synchronization status (Not Started/In Progress/Completed) of this stage
      When the task reaches the incremental phase, it will normally remain in the in-progress state without a specific end time.
      Meanwhile, the task business latency indicator is displayed on the right side of the status, refreshed automatically every 5 seconds.
      Note:
      The incremental phase may exhibit some business latency, which is normal. However, if the latency is significant, please monitor its impact on the business.

      Synchronization Overview

      The synchronization overview allows you to view the total data synchronization of real-time tasks according to different time dimensions and intervals:
      Parameters
      Description
      Time period
      You can select the last hour, today, this week, from the start of operation, or from a custom time range
      Total Reads
      The actual number of records read from the source during the execution of this task. This metric does not include the total amount of data eliminated through filtering or other methods.
      Total Successful Writes
      The total amount of data successfully written to the target during the execution of this task.
      Total Dirty Data
      The total amount of data that failed to be written to the target during the execution of this task. This metric does not include data that was not written due to active ignore/filter settings in the task configuration. This includes specific stops, abnormal restarts, other running policies, and data filtering.
      Total Running Time
      The total running time accumulated since the task started, including any pause time.

      Read and Write Details

      Read and Write Details enable you to view the read and write speed of real-time tasks and latency-related metrics by different time dimensions and intervals.
      Parameters
      Description
      Time period
      You can select the last hour, today, this week, from the start of operation, or from a custom time range
      Read Speed
      Total number of reads within the statistical interval / statistical interval
      Read Throughput
      Total read amount within the statistical interval / statistical interval
      Write Speed
      Number of successful writes within the statistical interval / statistical interval
      Write Throughput
      Total successful writes within the statistical interval / statistical interval
      Write Latency
      Link latency from Source end to Sink end, where write latency = System Time - Record Read Time (LatencyMarker timestamp on the reading end)
      Number of Job Restarts
      Current number of job restarts
      Log position information
      Real-time task CDC log position information
      Job Transmission Latency
      Time when data is read by Source (FetchTime) - Data event time (EventTime).
      Job Business Latency
      Time when data leaves Source (EmitTime) - Data event time (EventTime)
      Backpressure Metrics
      This metric reflects the relative speed of the reading end and writing end. Below 0.5% indicates the reading end is slower than the writing end; above 0.5% indicates the reading end is faster than the writing end

      Resource Metrics

      CheckPoint

      Metric Parameters
      Description
      Sample value
      Last Checkpoint Size
      The size of the most recent checkpoint for the current job
      751321.00Bytes
      Checkpoint Duration
      The time taken for the current job's checkpoint
      723.00ms
      Checkpoint Failure Count
      The total number of checkpoint failures for the current job
      8 times

      TaskManager

      Metric Parameters
      Description
      Sample value
      TaskManager CPU utilization
      The CPU utilization of the TaskManager for the current job.
      7.12%
      The amount of heap memory used by the TaskManager.
      The amount of heap memory used by the TaskManager for the current job.
      1040001560.00Bytes
      TaskManager Old Generation GC Count
      The number of old generation GCs in the TaskManager for the current job.
      3 times
      TaskManager Old Generation GC Time
      The total time spent on old generation GCs in the TaskManager for the current job.
      701.00ms
      The total physical memory usage (RSS) of the TaskManager.
      The total physical memory usage (RSS) of the JVM for the TaskManager in the current job, including heap, off-heap, native, and other memory areas.
      3597035110.00Bytes

      JobManager

      Metric Parameters
      Description
      Sample value
      JM CPU Load
      The recent CPU utilization of the JVM for the TaskManager.
      12%
      JM Head Memory
      The current heap memory usage of the TaskManager.
      1 time
      JM GC Count
      Status.JVM.GarbageCollector at the TaskManager level.<GarbageCollector>.Count, GC (garbage collection) count.
      5 times
      JM GC Time
      Status.JVM.GarbageCollector at the TaskManager level.<GarbageCollector>.Time, GC (garbage collection) time.
      65ms

      Link Details

      Maintenance support for whole database synchronization tasks includes viewing synchronization link details. Click Real-time Operation and Maintenance > Task Name > Statistics Metrics > Link Details to access the link details maintenance page. Link details allow you to view data read/write metrics at the table granularity.

      Read End

      Source Table Metrics: Shows detailed statistics of tables successfully read in whole database synchronization tasks.
      Including database name, table name, read rows/bytes, read speed/throughput.
      Trend charts support viewing detailed metrics of single tables during task execution.
      Search by table name or database name is supported.
      Trend Chart
      By default, the trend chart of various metrics in the past hour is expanded, and the display period can be adjusted manually.
      Data in the pop-up window is automatically refreshed every minute, and manual refresh is supported.

      Write End

      Target Table Metrics: Shows table statistics for whole database tasks.
      Including database name, table name, write rows/bytes, write speed/throughput, dirty data.
      Trend charts support viewing detailed metrics of single tables during task execution.
      Search by table name or database name is supported.
      Trend Chart
      By default, the trend chart of various metrics in the past hour is expanded, and the display period can be adjusted manually.
      Data in the pop-up window is automatically refreshed every minute, and manual refresh is supported.
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy