tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Data Development and Governance Platform
    Documentation
    Download PDF

    Product Overview

    Last updated: 2024-07-15 17:16:52
    Download PDF
      Data Development and Governance Platform WeData (hereinafter referred to as WeData) is a cloud-based one-stop data development and governance platform, integrating a full-link DataOps data development capability that includes data integration, data development, and task operations, as well as a series of data governance and operation capabilities such as data maps, data quality, and data security. It aims to help enterprises reduce costs and increase efficiency in data construction and application, maximizing the data value.

      Positioning

      Target Industries and Users

      Applicable to many industries including government, finance, pan-internet, industry, energy, transportation, education, cultural tourism, real estate, retail, health care, and media. The audience includes, but is not limited to:
      Technical personnel engaged in data development, algorithm development, and data operations.
      Business personnel engaged in data analysis and product operation.
      Management personnel responsible for data security compliance.
      Management personnel who control the company's core data assets.

      Business Challenges and Pain Points

      Since the explosion of the information technology revolution, and with the recent rapid development of the mobile internet, along with the continuous deepening evolution and implementation of the Internet+ concept, enterprises across various industries have accumulated more and more data, thereby deriving an urgent need for data processing and application. However, this process also faces many problems and challenges:
      Complex infrastructure construction: A dazzling array of big data technologies such as Hadoop, Spark, etc., make construction complex.
      Weak technical risk resistance: Disjointed development and testing, high probability of data errors, numerous data tasks, complex dependencies, and lack of effective change control.
      Complex data links: Open source projects often solve specific scenarios, requiring combining multiple open source projects to build complete data links.
      Difficult data management: Involves cross-departmental and cross-team collaboration, complex team roles, and high communication costs.
      Difficult data governance implementation: Data quality, data security, etc., cannot be guaranteed, and upper-level applications dare not use data confidently.
      Long business construction period: Data warehouse construction cycle is too long, taking half a year or even a year; slow response to data needs, with two to three days of delayed response.

      Core Capabilities

      WeData offers comprehensive product services for data production and consumption, with key service capabilities as follows:

      Collaboration

      Based on the collaborative space around the data value chain, different roles in the data team can collaborate better, breaking down the silos between teams and shortening the path from raw data to data value.
      DataOps Philosophy
      In large-scale task development scenarios, it enables high-concurrency online execution of data development and testing.
      Developers focus on task development and unit testing, avoiding the learning cost of business logic.
      Orchestration personnel focus on task orchestration and scheduling configuration, with dedicated personnel for specific tasks to shorten the implementation cycle.
      In agile development scenarios, integration of development and orchestration can improve efficiency.
      During the process of implementing orchestration business logic, data task development is completed.
      Allows simultaneous testing of data logic and business logic.
      Implementation Process
      Develop first, orchestrate later: Workflow design does not block development work, developers do not need to understand orchestration logic.
      After completing the development space, import into the orchestration space, with dedicated personnel for task orchestration.
      Suitable for centralized teams with large-scale and high-concurrency development tasks.
      Orchestrate first, develop later: Developers understand the business logic, design workflows first, then develop.
      Directly orchestrate and develop test tasks in the orchestration space, making it more agile.
      Suitable for small-scale or incremental tasks agile development mode for subteams.

      Efficiency

      Based on DataOps agile iteration, automated processes, and tools to enhance data reliability, it can accelerate the efficiency of data production and link analysis.
      Agile and Easy to Use: Supports incremental code development and release; supports automatic code completion; offers visual drag and drop for process design; supports online code debugging and log viewing.
      Flexible Development: The development mode is adaptable to multiple scenarios, supporting both development before orchestration and orchestration before development.
      High Performance and Scalability: A high-performance scheduling engine supports ten million tasks scheduling per day, can interface with various engines and support engine extensions, and supports more than 20 JDBC interface engines, including EMR, DLC, TBDS, RDS, and others by default.
      DataOps Philosophy
      Supports version management capabilities such as submission, comparison, and rollback to support the gray release of tasks.
      Supports the incremental release of tasks, events, parameters, functions, rather than the traditional periodic release.
      Agile development, rapid iteration, to shorten the overall data assetization cycle.
      Implementation Process
      After data task development is complete, version submission is required to reflect in the workflow.
      Different version tasks can be quickly debugged in the same workflow.
      Different projects with the same workflow based on different task versions achieve gray release.
      Incremental releases are done by date in release management, allowing for rapid iterations.

      Integration

      Serves multiple roles in enterprise data management, data production, data application, and data operations, providing an integrated product experience from different perspectives.
      End-to-end Production Governance: Provides strong quality and security assurance for data production and consumption through pre-planning, in-process exception blocking, post-event quality and cost analysis, and secure control over data circulation.
      One-stop Operational Governance: Based on the concepts of data self-service and democratization, it makes searching, understanding, analyzing, and sharing data easier on a stable and secure basis, through data mapping, insights, and sharing.

      Quality

      Data quality control throughout the pre-, during, and post-phases, integrated into the DataOps pipeline development process to ensure comprehensive data quality improvement.
      DataOps Philosophy
      Transitioning from post-event quality scoring to in-process quality monitoring, integrated testing comprises both code testing and data testing to ensure high-quality data analysis.
      Transitioning from post-event standard benchmarking to pre-event standard implementation to ensure data quality and consistency in statistical caliber during data analysis.
      Implementation Process
      Data tasks/workflows require online debugging before submission, automatically initiating quality monitoring tasks for corresponding data tables.
      Agile data warehouse modeling tools support direct referencing of pre-defined data standards during modeling to ensure early compliance and prevent failure at the source.
      Tables following data standards support setting a zero-tolerance threshold for dirty data during data integration tasks to ensure compliance.
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy