In the era of big data, the emergence of massive amounts of data has brought new opportunities to enterprises, as well as complex management challenges. How to efficiently use data resources, reduce operation and maintenance costs, and improve system performance has become a common problem for every enterprise. Tencent Cloud's Big Data Intelligent Steward, driven by AI, adopts a brand-new method to provide enterprises with comprehensive solutions from data insights to intelligent governance through real-time exception identification and prediction, multi-dimensional policy insights, in-depth AI analysis, cost and performance optimization, and automated governance, thus becoming a powerful helper in the field of big data management.

1、Use AI to drive efficient management and solve big data dilemmas from the source.

The core value of the intelligent steward lies in simplifying complex big data management tasks by intelligent means. The traditional manual Ops mode is inefficient and costly, while the intelligent steward replaces manual operation with mature AI technology, making problem detection and resolution faster and more accurate.

The system can monitor the running status of cluster in real time, identify exceptions, predict faults, perform root cause analysis, and improve the timeliness of problem handling to a new level. At the same time, the intelligent steward can also provide optimization suggestions from multiple dimensions to help enterprises maximize resource utilization efficiency.

The panoramic architecture of the intelligent steward includes five modules: data access, data processing and storage, data identification and insight, intelligent AI predictive analysis, and automated governance, covering six levels including basic monitoring, computing, storage, resources, logs, and events. Through precise diagnosis of data across domains, multi-level linkage root cause analysis can be implemented to solve problems more efficiently.

2、Intelligent functional modules empower data management

The core functional modules of the intelligent steward include:

➢ Computing Insight: Identify issues such as large queries, full table scans, small file tables, data skew, and improper resource allocation by monitoring and analyzing queries in real time, so as to optimize the usage efficiency of computing resources.

➢ Storage insight: Identify small files, small file storage tables and the distribution of storage data, and provide suggestions on reasonable partition, merge, compression and hierarchical storage to improve storage performance and query timeliness.

➢ Basic Data Characteristics: Identify key factors that affect system performance by using data mining and feature extraction technologies, and detect exceptions in a timely manner.

➢ AI Failure Prediction: Predict potential failures by constructing machine learning models to analyze historical failure data, and provide preventive maintenance suggestions to reduce system downtime.

3、Best practices, significant results

Query optimization for e-commerce enterprises

An e-commerce company has long been troubled by the query efficiency decline and long-term resource occupation caused by small files. With storage and computing insight technology, the intelligent steward has successfully reduced the quantities of small files from over 100 million to 36 million, with a governance percentage of 64% and a query efficiency increase of 26%. System pressure has been relieved, and the execution efficiency of key tasks has greatly improved.

Storage management for marketing solution enterprises

A marketing solution provider used the intelligent steward's storage insight feature to clean up a large number of small files and optimize the distribution of hot and cold data tables, successfully saving more than 30% of storage space while greatly improving query response speed.

Accurate prediction of cluster exceptions

In cluster node monitoring, the intelligent steward identifies more than 30 exceptions every day through digital feature analysis, shortens the detection time to seconds, and predicts the exception trend in the next 1-3 days. Timely preventive measures are taken to greatly reduce the risk of system failure and ensure the stable operation of the cluster.

4、From Management to Insights, Let Intelligence Lead the Future

Tencent Cloud Big Data Intelligent Steward has become an important part of Tencent Cloud's big data management ecosystem. It has been fully integrated with core products such as Elastic MapReduce and Data Lake Compute DLC. Subsequently, it will continue to undertake other big data products to achieve intelligent in-depth insights. Through data feature recognition, rich exception scenarios, and root cause algorithm support, it realizes in-depth insights and intelligent management of resource, service and business scheduling data. We believe that with the continuous development of AI and other technologies, the intelligent steward will be able to better serve users, solve various challenges they face, and promote the development of enterprise data management in an intelligent, efficient and low-cost direction, making the intelligent steward a reliable partner for enterprise digital and intelligent transformation.