tencent cloud

All product documents
Elastic MapReduce
Hive Data Table Analysis
Last updated: 2024-10-30 10:43:13
Hive Data Table Analysis
Last updated: 2024-10-30 10:43:13

Feature Introduction

Provides data distribution and trend information based on the storage volume of databases and data tables.
Provides the distribution of data tables based on their last access time, serving as a reference for hot and cold data distribution.
The proportion of small files at the data table level and the data volume at the partition level within tables can help identify issues such as small files and partition data skew.

Directions

1. Log in to the EMR console, and click the corresponding Cluster ID/Name in the cluster list to enter the cluster details page.
2. On the cluster details page, click Cluster Services, then select Hive component and select Operations > Data Table Analysis in the top-right corner. This provides the relevant data table and data information collected based on the Hive MetaStore up to the last collection point.
3. Statistical Graph
3.1 Statistics view View metrics such as the Hive database, total number of tables, total storage volume, and daily increase and daily comparison information for related metrics.
3.2 See the last access time of data tables to view the distribution, which can be used as a reference for hot and cold data distribution.
Note:
Interval
Description
Within 3 months
Most recent access time less than 3 months
3 months to 1 year
Most recent access time greater than or equal to 3 months and less than or equal to 1 year
1 year to 5 years
Most recent access time greater than 1 year and less than or equal to 5 years
Over 5 years
Most recent access time greater than 5 years
Others
Collection not enabled at the COS/CHDFS side
Most recent access time for stored data not collected
4. The trend view provides the historical growth trend of the number of databases, tables, and table storage size. It includes a dimension for tables with small files, allowing observation of the distribution and growth of small file tables.
5. You can view the database to which a table belongs, storage size, number of files, proportion of small files, and the table’s partition details. The cloud file storage size and the proportion of small files provide an intuitive view of Hive small file issues.
6. Click Operation to view partitions, where you can see information such as partition name, partition size, and total number of files. Partition size and total number of files help identify partition skew and file quantity details.
Risk note
The analysis data for data table analysis will be collected daily starting at 14:00 UTC+8.
1. It mainly collects HMS database and table, partition metadata, and NameNode directory information. This operation will slightly increase the request load for HMS and NameNode. If the request volume continuously exceeds the load capacity, you can ticket feedback to disable this feature.
2. HMS and NameNode data collection only involves metadata and does not affect specific business data.

Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No

Feedback

Contact Us

Contact our sales team or business advisors to help your business.

Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 available.

7x24 Phone Support