tencent cloud

Feedback

Solutions for High Disk Space Utilization

Last updated: 2024-05-07 16:17:38

    Overview

    The disk in MongoDB mainly stores data and indexes, as well as some system files and log files. Disk space usage rate is a very important monitoring metrics. When disk space is fully used, the MongoDB instance will be unable to continue writing new data, leading to instance failure and shutdown. Therefore, monitoring disk usage and promptly taking measures to free up disk space is key to ensuring normal operation of the MongoDB instance.

    Viewing Disk Space Usage

    Quickly Viewing Monitoring Metrics

    Log in to the MongoDB console. On the System Monitoring tab, you can view the trend view of Disk Space Usage Rate. For specific operations, see View Monitoring Data.
    

    Detailed Analysis of Disk Space Usage

    Log in to the MongoDB console. In the left sidebar, choose Performance Optimization, and then choose the Space Analysis tab. Through the Space Analysis of DBbrain (TencentDB for DBbrain), you can further analyze the details of disk space usage for the database, including database collection space, index space, physical file size, database size, data proportion, number of rows in collections, and other analytical data and views. For specific operations, see Space Analysis.
    
    Use MongoDB's own commands db.stats() and db.$collection_name.stats() to analyze disk space usage. For detailed information, see the following table.
    Analysis Command
    Command Definition
    This command is used to retrieve the statistical information of the current database. Executing the db.stats() command returns a document containing various information about the current database. For example: database name, data size, index size, number of collections, etc.
    This command is used to retrieve the statistical information of a specified collection. Executing the db.collection.stats() command returns a document containing various information about the specified collection. For example: collection name, number of documents, data size, number of indexes, etc.
    This command is used to retrieve the storage space size occupied by a specified collection. Executing the db.collection.storageSize() command returns the storage space size occupied by the specified collection, in bytes. The storage space size returned by this command includes the space occupied by data and indexes in the collection, but does not include other overheads of the MongoDB instance, such as log files and temporary files.
    This command is used to retrieve the storage space size occupied by all indexes of a specified collection. Executing the db.collection.totalIndexSize() command returns the storage space size occupied by all indexes of the specified collection, in bytes. The storage space size returned by this command does not include the space occupied by data in the collection, only the space occupied by all indexes of the collection.
    This command is used to retrieve the total storage space size occupied by a specified collection. Executing the db.collection.totalSize() command returns the total storage space size occupied by the specified collection, in bytes. The storage space size returned by this command includes the space occupied by data and indexes in the collection, as well as other overheads of the MongoDB instance, such as log files and temporary files.

    Problem Analysis

    TencentDB for MongoDB by default uses the WiredTiger engine. When documents are deleted, disk space is not directly reclaimed. However, when new data is inserted, MongoDB will reuse the previously occupied space instead of consuming additional new disk space. As more deletions occur, fragmentation will increase. The following code allows you to view the fragmentation rate of all collections in a specified database at once.
    When the disk space usage rate of the instance reaches 80% to 85% or higher, the risk of running out of space can be avoided by either reducing the actual space occupied by the database or expanding the storage space.
    ## Creating a Function for Querying the Fragmentation Rates
    function getCollectionDiskSpaceFragRatio(dbname, coll) { var res = db.getSiblingDB(dbname).runCommand({ collStats: coll }); var totalStorageUnusedSize = 0; var totalStorageSize = res['storageSize'] + res['totalIndexSize']; Object.keys(res.indexDetails).forEach(function(key) { var size = res['indexDetails'][key]['block-manager']['file bytes available for reuse']; print("index table " + key + " unused size: " + size); totalStorageUnusedSize += size; }); var size = res['wiredTiger']['block-manager']['file bytes available for reuse']; print("collection table " + coll + " unused size: " + size); totalStorageUnusedSize += size; print("collection and index table total unused size: " + totalStorageUnusedSize); print("collection and index table total file size: " + totalStorageSize); print("Fragmentation ratio: " + ((totalStorageUnusedSize * 100.0) / totalStorageSize).toFixed(2) + "%"); }
    ## Specifying a Database for Querying the Fragmentation Rates of All Collections use xxxdb db.getCollectionNames().forEach((c) => {print("\\n\\n" + c); getCollectionDiskSpaceFragRatio(db.getName(), c);});

    Solutions

    High Disk Usage Rate and High Fragmentation Rates

    For TencentDB for MongoDB versions 4.4 and later.
    If the disk usage rate is high (generally exceeding 80% to 85%), and the fragmentation rates are also high (where it becomes beneficial to perform space reclamation if they exceed 25%), for TencentDB for MongoDB running on a replica set architecture with versions 4.4 and later, use the command db.runCommand({compact:"collectionName"}) to compress documents within the specified collection, thereby releasing disk space. In this command, replace collectionName with the actual name of the targeted collection.
    Note:
    When MongoDB performs a Compact operation, it compresses the whole database. Thus, during this process, operations like creating and deleting collections, as well as creating and deleting indexes, will be temporarily blocked. Meanwhile, other operations, such as querying, will not be impacted directly, but there may be performance implications leading to increased latency in requests. It is recommended to execute this operation during business off-peak hours.
    For TencentDB for MongoDB versions earlier than 4.4.
    If the disk usage rate is high (generally exceeding 80% to 85%), and the fragmentation rate is also high (where it becomes beneficial to perform space reclamation if they exceed 25%), for TencentDB for MongoDB running versions below 4.4, it is not recommended to execute the compact operation. This is because doing so will block all incoming requests to the instance and might potentially result in issues where the Compact process does not effectively manage indexes. The solutions are as follows:
    Upgrade the version. It is recommended to upgrade to the latest version for better performance and stability. For specific operations, see Version Upgrade.
    If you choose not to upgrade, you can perform a logical migration to rebuild nodes in order to attain space shrinkage. This process may result in multiple temporary disconnections. For specific operations, contact after-sales support or Submit a Ticket.

    High Disk Usage Rate but Low Fragmentation Rates

    If the disk usage rate is high (generally exceeding 80% to 85%), but the fragmentation rate is relatively low (less than 20%), it is not recommended to perform the campact operation, as MongoDB inherently reuses this space. In such circumstances, consider expanding the disk space. For specific operations, see Change Mongod Node Configuration Specifications.
    
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support