tencent cloud

Feedback

Supported Monitoring Indicators

Last updated: 2024-07-08 15:56:51

    Overview

    You can view the metric monitoring data in real time to stay up to date with the metric performance of running tasks.
    Statistical dimension (data migration: app_id, migratejob_id; data sync: appid, replicationjobid; data subscription: appId, subscribeid)
    Statistical period: 60s, 300s.
    BPS: Bits per second, indicating the volume of data transferred per second.
    RPS: Rows per second, indicating the number of rows transferred per second.

    Application Scope

    Monitoring metrics can be displayed for the migration and sync links of MySQL/MariaDB/Percona/TDSQL-C for MySQL/TDSQL for MySQL.
    Monitoring metrics can be displayed for the subscription links of MySQL/MariaDB/Percona/TDSQL-C for MySQL/TDSQL for MySQL/TDSQL for PostgreSQL

    Data Migration

    MySQL/MariaDB/Percona/TDSQL-C for MySQL

    Metric Category
    Metric Name
    Parameter
    Unit
    Description
    BPS – Full Stage
    Full Export BPS
    MigrateDumperBps
    MB/s
    Volume of data exported from the source instance per second in the full stage
    Full Import BPS
    MigrateLoaderBps
    MB/s
    Volume of data imported to the target instance per second in the full stage
    BPS – Incremental Stage
    Incremental Export BPS
    MigrateRiverBps
    MB/s
    Volume of data exported from the source instance per second in the incremental stage
    Incremental Import BPS
    MigrateSinkerBps
    MB/s
    Volume of data imported to the target instance per second in the incremental stage
    RPS – Full Stage
    Full Export RPS
    MigrateDumperRps
    Count/s
    Number of rows of data exported from the source instance per second in the full stage
    Full Import RPS
    MigrateLoaderRps
    Count/s
    Number of rows of data imported to the target instance per second in the full stage
    RPS – incremental stage
    Source Instance Data Extraction RPS (Incremental Export RPS)
    MigrateCaptureRps
    Count/s
    This metric is only used in the transitory stage, and the "Incremental Export RPS" metric will be finally used.
    Incremental Export RPS
    MigrateRiverRps
    Count/s
    Number of rows of data exported from the source instance per second in the incremental stage
    Target Instance Data Loading RPS (Incremental Import RPS)
    MigrateLoadRps
    Count/s
    This metric is only used in the transitory stage, and the "Incremental Import RPS" metric will be finally used.
    Incremental Import RPS
    MigrateSinkerRps
    Count/s
    Number of rows of data imported to the target instance per second in the incremental stage
    Network Latency – Full Stage
    Source Instance Network Latency During Full Export
    MigrateDumperNetworkLag
    ms
    Network latency between data export and source instance in the full stage
    Target Instance Network Latency During Full Import
    MigrateLoaderNetworkLag
    ms
    Network latency between data import and target instance in the full stage
    Network Latency – Incremental Stage
    Source Instance Network Latency During Incremental Export
    MigrateRiverNetworkLag
    ms
    Network latency between data export and source instance in the incremental stage
    Target Instance Network Latency During Incremental Import
    MigrateSinkerNetworkLag
    ms
    Network latency between data import and target instance in the incremental stage
    Data Migration Time Lag
    MigrateLag
    s
    Time lag between source and target instances in the incremental stage
    Calculation method: Time lag = the current time of the source instance - the time recorded when the latest source instance binlog event is being executed in the target instance. Calculation of the data migration time lag depends on the incremental binlogs of the source database. Therefore, if there are no DDL or DML operations performed on the source database for a long time, the value of this metric will gradually increase and thus cannot reflect the actual migration time lag. If the value is -1, it indicates that the existing data has been migrated, but there is no refresh of the incremental data. In this case, you can refresh the metric by running a SQL statement on the source database to get the correct metric value.
    Data Gap in Data Migration
    MigrateLagData
    MBytes
    Data gap between the source and target instances in the incremental stage
    Calculation method: Data gap = the file offset of the latest source instance binlog event - the file offset of the latest source instance binlog event which is being executed in the target instance. If the two offsets are in different binlog files, this value will be an estimate. If there are no DDL or DML operations performed on the source database for a long time, the value of this metric will gradually increase and thus cannot reflect the actual sync data gap. If the value is -1, it indicates that the existing data has been synced, but there is no refresh of the incremental data.
    RPS Utilization
    RPS Utilization During Incremental Import
    MigrateSinkerRpsUsage
    %
    RPS utilization = real-time RPS/maximum RPS. The maximum RPS depends on the specification you select when configuring the link.
    When the RPS utilization almost reaches 100%, the incremental migration can no longer be accelerated. You need to upgrade the specification for faster migration speed.
    When the link is configured with the highest specification, there is no RPS limit, and the RPS may exceed 100%.

    TDSQL for MySQL

    Currently, only the following incremental migration metrics are supported.
    Metric Name
    Parameter
    Unit
    Description
    Source Instance Data Extraction RPS
    MigrateCaptureRps
    Count/s
    Number of rows of source instance data read by DTS per second
    Target Instance Data Loading RPS
    MigrateLoadRps
    Count/s
    Number of rows of data migrated to the target instance per second
    Data Migration Latency
    MigrateLag
    s
    Time lag between source and target instances
    Calculation method: Time lag = the current time of the source instance - the time recorded when the latest source instance binlog event is being executed in the target instance.
    Calculation of the data migration time lag depends on the incremental binlogs of the source database. Therefore, if there are no DDL or DML operations performed on the source database for a long time, the value of this metric will gradually increase and thus cannot reflect the actual migration time lag. If the value is -1, it indicates that the existing data has been migrated, but there is no refresh of the incremental data. In this case, you can refresh the metric by running a SQL statement on the source database to get the correct metric value.
    Data Gap in Data Migration
    MigrateLagData
    MBytes
    Data gap between the source and target instance
    Calculation method: Data gap = the file offset of the latest source instance binlog event - the file offset of the latest source instance binlog event which is being executed in the target instance. If the two offsets are in different binlog files, this value will be an estimate.
    If there are no DDL or DML operations performed on the source database for a long time, the value of this metric will gradually increase and thus cannot reflect the actual sync data gap. If the value is -1, it indicates that the existing data has been migrated, but there is no refresh of the incremental data.

    Data Sync

    MySQL/MariaDB/Percona/TDSQL-C for MySQL

    Metric Category
    Metric Name
    Parameter
    Unit
    Description
    BPS – Full Stage
    Full Export BPS
    ReplicationDumperBps
    MB/s
    Volume of data exported from the source instance per second in the full stage
    Full Import BPS
    ReplicationLoaderBps
    MB/s
    Volume of data imported to the target instance per second in the full stage
    BPS – Incremental Stage
    Incremental Export BPS
    ReplicationRiverBps
    MB/s
    Volume of data exported from the source instance per second in the incremental stage
    Incremental Import BPS
    ReplicationSinkerBps
    MB/s
    Volume of data imported to the target instance per second in the incremental stage
    RPS – Full Stage
    Full Export RPS
    ReplicationDumperRps
    Count/s
    Number of rows of data exported from the source instance per second in the full stage
    Full Import RPS
    ReplicationLoaderRps
    Count/s
    Number of rows of data imported to the target instance per second in the full stage
    RPS – incremental stage
    Source Instance Data Extraction RPS (Incremental Export RPS)
    ReplicationCaptureRps
    Count/s
    This metric is only used in the transitory stage, and the "Incremental Export RPS" metric will be finally used.
    Incremental Export RPS
    ReplicationRiverRps
    Count/s
    Number of rows of data exported from the source instance per second in the incremental stage
    Target Instance Data Loading RPS (Incremental Import RPS)
    ReplicationLoadRps
    Count/s
    This metric is only used in the transitory stage, and the "Incremental Import RPS" metric will be finally used.
    Incremental Import RPS
    ReplicationSinkerRps
    Count/s
    Number of rows of data imported to the target instance per second in the incremental stage
    Network Latency – Full Stage
    Source Instance Network Latency During Full Export
    ReplicationDumperNetworkLag
    ms
    Network latency between data export and source instance in the full stage
    Target Instance Network Latency During Full Import
    ReplicationLoaderNetworkLag
    ms
    Network latency between data import and target instance in the full stage
    Network Latency – Incremental Stage
    Source Instance Network Latency During Incremental Export
    ReplicationRiverNetworkLag
    ms
    Network latency between data export and source instance in the incremental stage
    Target Instance Network Latency During Incremental Import
    ReplicationSinkerNetworkLag
    ms
    Network latency between data import and target instance in the incremental stage
    Data Sync Time Lag
    DtsReplicationLag
    s
    Sync time lag between source and target instances in the incremental stage
    Calculation method: Time lag = the current time of the source instance - the time recorded when the latest source instance binlog event is being executed in the target instance. Calculation of the data sync time lag depends on the incremental binlogs of the source database. Therefore, if there are no DDL or DML operations performed on the source database for a long time, the value of this metric will gradually increase and thus cannot reflect the actual sync time lag. If the value is -1, it indicates that the existing data has been synced, but there is no refresh of the incremental data. In this case, you can refresh the metric by running a SQL statement on the source database to get the correct metric value.
    Data Gap in Data Sync
    DtsReplicationLagData
    MBytes
    Sync data gap between the source and target instances in the incremental stage.
    Calculation method: Data gap = the file offset of the latest source instance binlog event - the file offset of the latest source instance binlog event which is being executed in the target instance. If the two offsets are in different binlog files, this value will be an estimate. If there are no DDL or DML operations performed on the source database for a long time, the value of this metric will gradually increase and thus cannot reflect the actual sync data gap. If the value is -1, it indicates that the existing data has been synced, but there is no refresh of the incremental data.
    RPS utilization
    RPS Utilization During Incremental Import
    ReplicationSinkerRpsUsage
    %
    RPS utilization = real-time RPS/maximum RPS. The maximum RPS depends on the specification you select when configuring the link.
    When the RPS utilization almost reaches 100%, the incremental migration can no longer be accelerated. You need to upgrade the specification for faster migration speed.
    When the link is configured with the highest specification, there is no RPS limit, and the RPS may exceed 100%.

    TDSQL for MySQL

    Currently, only the following incremental sync metrics are supported.
    Metric Name
    Parameter
    Unit
    Description
    Source Instance Data Extraction RPS
    ReplicationCapture_rps
    Count/s
    Number of rows of source instance data read by DTS per second
    Target Instance Data Loading RPS
    ReplicationLoadRps
    Count/s
    Number of rows of data migrated to the target instance per second
    Data Sync Time Lag
    DtsReplicationLag
    s
    Time lag between source and target instances
    Calculation method: Time lag = the current time of the source instance - the time recorded when the latest source instance binlog event is being executed in the target instance.
    Calculation of the data sync time lag depends on the incremental binlogs of the source database. Therefore, if there are no DDL or DML operations performed on the source database for a long time, the value of this metric will gradually increase and thus cannot reflect the actual sync time lag. If the value is -1, it indicates that the existing data has been migrated, but there is no refresh of the incremental data. In this case, you can refresh the metric by running a SQL statement on the source database to get the correct metric value.
    Data Gap in Data Sync
    DtsReplicationLagData
    MBytes
    Data gap between the source and target instance
    Calculation method: Data gap = the file offset of the latest source instance binlog event - the file offset of the latest source instance binlog event which is being executed in the target instance. If the two offsets are in different binlog files, this value will be an estimate.
    If there are no DDL or DML operations performed on the source database for a long time, the value of this metric will gradually increase and thus cannot reflect the actual sync data gap. If the value is -1, it indicates that the existing data has been synced, but there is no refresh of the incremental data.

    Data Subscription

    MySQL/TDSQL-C for MySQL

    Metric Category
    Metric Name
    Parameter
    Unit
    Description
    Data Production
    GTID Quantity Gap Between Subscription Service and Source Database
    ProducerLag
    Count
    Gap between the number of GTIDs in the binlog event already parsed by the data subscription service and the number of GTIDs in the latest binlog event generated by the source database.
    If there are no DDL or DML operations performed on the source database for a long time, the value of this metric will gradually increase and thus cannot reflect the actual sync time lag. If the value is -1, it indicates that there is no refresh of the incremental data.
    Number of Transactions Parsed per Second
    ProducerTps
    Count/s
    Number of transactions parsed from the source instance binlog per second in the incremental stage
    Incremental Export BPS
    SubscribeRiverBps
    MB/s
    Volume of data exported from the source instance per second in the incremental stage
    Incremental Import BPS to Kafka
    SubscribeSinkerBps
    MB/s
    Volume of data imported to the built-in Kafka of the subscription service per second in the incremental stage
    Data Consumption
    Subscription Partition Consumption Gap
    SubscribePartitionLag
    Count
    Gap between the offset of data to be consumed and that of consumed data. The statistical dimensions are "Partition", "Group", and "Task".
    Subscription Partition Consumption Time Lag
    SubscribePartitionLagTime
    s
    Time lag between the consumed data and the source instance. The statistical dimensions are "Partition", "Group", and "Task".

    MariaDB/Percona/TDSQL for MySQL

    Metric Name
    Parameter
    Unit
    Description
    GTID Quantity Gap Between Subscription Service and Source Database
    ProducerLag
    Count
    Gap between the number of GTIDs in the binlog event already parsed by the data subscription service and the number of GTIDs in the latest binlog event generated by the source database.
    If there are no DDL or DML operations performed on the source database for a long time, the value of this metric will gradually increase and thus cannot reflect the actual sync time lag. If the value is -1, it indicates that there is no refresh of the incremental data.
    Number of Transactions Parsed per Second
    ProducerTps
    Count/s
    Number of transactions parsed from the source instance binlog per second

    TDSQL for PostgreSQL

    Metric Name
    Parameter
    Unit
    Description
    LSN Gap Between Subscription Service and Source Database
    ProducerLsnLag
    MBytes
    LSN gap between the offset of parsed logs and that of the latest logs
    Number of Transactions Parsed per Second
    ProducerTps
    Count/s
    Number of transactions parsed from the source instance binlog per second

    Viewing Monitoring Metrics

    1. Log in to the DTS console and select Data Migration, Data Sync, or Data Subscription on the left sidebar as needed.
    2. You can view monitoring metrics in the two ways below:
    Option 1: Select the target migration task and click the
    
    icon to enter the monitoring view. You can also click the
    
    icon to view the detailed metric monitoring information.
    Option 2: Click the ID of the target migration task to enter the task details page.
    Select the Monitoring Data tab to view the metric monitoring data.
    3. Select a time range.
    You can select a custom time range or select from the available time ranges quickly.
    Three time dimensions are supported for comparison: week-over-week, day-over-day, and custom date comparison.
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support