tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Cloud Object Storage
    Documentation
    Download PDF

    Working with COSBench

    Last updated: 2024-11-06 18:06:09
    Download PDF

      COSBench Overview

      COSBench is an open-source stress test tool developed by Intel for testing the performance of cloud object storage systems. As a cloud storage system compatible with S3 protocol, COSBench can be used to perform benchmark tests on the read/write performance of COS.

      System Environment

      It is recommended that you run COSBench in CentOS 7.0 or a later version. If you run it in Ubuntu, unexpected issues may occur.

      Performance Factors

      CPU cores: a small number of CPU cores coupled with a large number of running workers is very likely to cause high overheads for context switching. Therefore, 32 or 64 cores are recommended for the test.
      NIC: the outbound traffic from the server is limited. To test the traffic for large files, an NIC above 10 GB is recommended.
      Network link: public network links vary in quality. Charges will incur for public network downstream traffic for downloads over public network. Therefore, private network is recommended for access within the same region.
      Test duration: in order to get a reliable value, we recommend a longer test period.
      Test environment: the version of the JDK running on your program is also a key performance factor. For example, when testing on HTTPS, with an earlier JDK version, GCM bugs may occur for the encryption algorithm, as well as the locking issues for the random number generator.

      Directions

      1. Download COSBench 0.4.2.c4.zip from GitHub and decompress it on your server.
      2. Install the COSBench dependent library and run the following command.
      For CentOS, run the following command to install the dependencies:
      sudo yum install nmap-ncat java curl java-1.8.0-openjdk-devel -y
      For Ubuntu, run the following command to install the dependencies:
      sudo apt install nmap openjdk-8-jdk
      3. Edit the file s3-config-sample.xml and configure a test job. The test job is divided into the following five stages.
      3.1 init: creates a bucket.
      3.2 prepare: uses parallel threads (or workers) to PUT (upload) objects with specified size to read in the main stage.
      3.3 main: uses parallel workers to read and write objects for a specified period of time.
      3.4 cleanup: deletes the created objects.
      3.5 dispose: deletes buckets.
      The sample configuration is as shown below.
      <?xml version="1.0" encoding="UTF-8" ?>
      <workload name="s3-50M-sample" description="sample benchmark for s3">
      
        <storage type="s3" config="accesskey=************************************;secretkey=YaWIuQmCSZ5ZMniUM6hiaLxHnxxxxxx;endpoint=http://cos.ap-beijing.myqcloud.com" />
      
        <workflow>
      
          <workstage name="init">
            <work type="init" workers="10" config="cprefix=examplebucket;csuffix=-1250000000;containers=r(1,10)" />
          </workstage>
      
          <workstage name="prepare">
            <work type="prepare" workers="100" config="cprefix=examplebucket;csuffix=-1250000000;containers=r(1,10);objects=r(1,1000);sizes=c(50)MB" />
          </workstage>
      
          <workstage name="main">
            <work name="main" workers="100" runtime="300">
              <operation type="read" ratio="50" config="cprefix=examplebucket;csuffix=-1250000000;containers=u(1,10);objects=u(1,1000)" />
              <operation type="write" ratio="50" config="cprefix=examplebucket;csuffix=-1250000000;containers=u(1,10);objects=u(1000,2000);sizes=c(50)MB" />
            </work>
          </workstage>
      
          <workstage name="cleanup">
            <work type="cleanup" workers="10" config="cprefix=examplebucket;csuffix=-1250000000;containers=r(1,10);objects=r(1,2000)" />
          </workstage>
      
          <workstage name="dispose">
            <work type="dispose" workers="10" config="cprefix=examplebucket;csuffix=-1250000000;containers=r(1,10)" />
          </workstage>
      
        </workflow>
      
      </workload>
      Parameter description
      Parameter
      Description
      accesskey, secretkey
      Key information. We recommend you use a sub-account key and follow the principle of least privilege to reduce risks. For information about how to obtain a sub-account key, see Access Key.
      cprefix
      Bucket name prefix, such as `examplebucket`.
      containers
      Value range for bucket names. A bucket name is made up of `cprefix` and `containers`, such as `examplebucket1` or `examplebucket2`.
      csuffix
      Your `APPID`. Note that `APPID` is prefixed with a -, such as `-1250000000`.
      runtime
      Duration of the stress test
      ratio
      Ratio of reads to writes
      workers
      Number of stress test threads
      4. Edit the file `cosbench-start.sh` and add the following parameter to the Java startup command line to disable S3 MD5 verification.```plaintext-Dcom.amazonaws.services.s3.disableGetObjectMD5Validation=true```![](https://qcloudimg.tencent-cloud.cn/raw/ac010bb86f091d709a0776b4e20a5858.png)5. Start the COSBench service. - For CentOS, run the following command:```plaintextsudo bash start-all.sh``` - For Ubuntu, run the following command:```plaintextsudo bash start-driver.sh &sudo bash start-controller.sh &```6. Run the following command to submit the job.```plaintextsudo bash cli.sh submit conf/s3-config-sample.xml```Check the test status at `http://ip:19088/controller/index.html` (replace the IP in this link with the IP of your own testing server).![](https://main.qcloudimg.com/raw/77f1631fa15141332d123fb472bab7ac.png)You can see the five work stages as shown below.![](https://main.qcloudimg.com/raw/3ccb5a60253ceb20c6da9292582c4355.png)7. The following example shows the performance tests of uploads and downloads on a CVM instance with 32 cores and 17 Gbps private network bandwidth in Beijing region. The test includes the following two stages. 1. prepare: 100 workers threads are run to upload 1,000 objects (50 MB each). 2. main: 100 workers read and write objects in parallel for 300 seconds.
      The test results after two stages above are as shown below.
      
      8. Run the following command to stop the test.
      sudo bash stop-all.sh
      
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy