tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Cloud Object Storage
    Documentation
    Download PDF

    Syncing Upload/Download/Copy - sync

    Last updated: 2024-11-01 20:59:19
    Download PDF

      Command Syntax

      The sync command is used to sync object upload, download, and copy. The difference between sync and cp is that sync first compares the CRC64 value of an object with the same name that already exists, and if the value is the same, the object will not be transferred.
      ./coscli sync <source_path> <destination_path> [flag]
      cp includes the following parameters:
      Parameter Format
      Description
      Example
      source_path
      Source file path, which can be a local path or a COS file path. The COS path is accessible by using the bucket alias or bucket name configured in the configuration file as detailed in Download and Installation Configuration. If you use the bucket name for access, you also need to include the endpoint flag.
      Local path: ~/example.txt
      COS file path specified with the bucket alias: cos://bucketalias/example.txt
      COS file path specified with the bucket name: cos://examplebucket-1250000000/example.txt
      destination_path
      Destination file path, which can be a local path or a COS file path. The COS path is accessible by using the bucket alias or bucket name configured in the configuration file as detailed in Download and Installation Configuration. If you use the bucket name for access, you also need to include the endpoint flag.
      Local path: ~/example.txt
      COS file path specified with the bucket alias: cos://bucketalias/example.txt
      COS file path specified with the bucket name: cos://examplebucket-1250000000/example.txt
      sync includes the following optional flags:
      Flag Abbreviation
      Flag Name
      Description
      None
      --include
      Includes specific objects.
      None
      --exclude
      Excludes specific objects.
      -r
      --recursive
      Specifies whether to traverse all objects in the directory recursively.
      None
      --storage-class
      Specifies the storage class of the uploaded file. Default value: STANDARD. For more information, see Storage Class Overview.
      None
      --part-size
      Part size. Default value: 32 MB; maximum value: 5 GB
      None
      --thread-num
      Number of concurrent threads. Default value: 5
      None
      --rate-limiting
      Speed limit for a single URL. Value range: 0.1–100 MB/s
      None
      --snapshot-path
      Specifies the directory where the snapshot information is stored when the uploaded or downloaded file is saved. During the next file upload or download, COSCLI will read the snapshot information in the specified directory for incremental upload or download. This option is used to speed up directory file sync.
      None
      --meta
      Metadata of the uploaded file, including certain HTTP standard attributes (HTTP Header) and custom metadata prefixed with x-cos-meta- (User Meta). The file metadata is in the format of header:value#header:value, such as Expires:2022-10-12T00:00:00.000Z#Cache-Control:no-cache#Content-Encoding:gzip#x-cos-meta-x:x.
      None
      --routines
      Specifies the number of files for concurrent upload or download threads between files. The default value is 3.
      None
      --fail-output
      This option determines whether to enable the error output of files when upload or download fails (The default value is true, enabled). If enabled, failed file transfers will be recorded in the specified directory (If no directory is specified, the default directory ./coscli_output is used). If disabled, only the number of failed files will be output to the console.
      None
      --fail-output-path
      This option is used to specify the error output folder for recording failed file uploads or downloads. By providing a custom folder path, you can control the location and name of the error output folder. If this option is not set, the default error log folder ./coscli_output is used.
      None
      --retry-num
      Number of frequency limit retry times (The default value is 0, indicating no retry will be performed). It can be specified as 1-10 times. When multiple machines execute download operations on the same COS directory simultaneously, you can specify this parameter to retry to avoid frequency limit errors.
      None
      --err-retry-num
      Number of error retry times (The default value is 0). Set it as 1-10 times, or set it to 0, indicating no retry will be performed.
      None
      --err-retry-interval
      Retry interval (only available when --err-retry-num is set as 1-10). Set the retry interval as 1-10 seconds. If it is not set or is set to 0, each retry interval will be a random value between 1-10 seconds.
      None
      --only-current-dir
      Whether to upload only files in the current directory, ignoring subdirectories and their contents (The default value is false, indicating they will not be ignored).
      None
      --disable-all-symlink
      Whether to ignore the subfiles and subdirectories of all the soft links during upload (The default value is true, indicating they will not be ignored).
      None
      --enable-symlink-dir
      Whether to upload the subdirectories of the soft links (The default value is false, indicating they will not be uploaded).
      None
      --disable-crc64
      Whether to disable CRC64 data validation (The default value is false, indicating validation is enabled).
      None
      --delete
      Deletes any other files in the specified target path, retaining only the files synchronized this time (The default value is false, indicating the files will not be deleted). It is recommended to enable version control before using the --delete option to prevent accidental data deletion.
      None
      --backup-dir
      Synchronizes the backup of deleted files, which preserves files that are deleted on the target end but do not exist on the source end (effective only during the download, and must be specified when --delete=true). For upload and bucket copies, use version control to restore accidentally deleted data.
      None
      --force
      Forced operation, without confirmation prompt (The default value is false).
      No
      --disable-checksum
      The default value is false, verifying the CRC64 of the entire file. If set to true, only verify shard CRC64.
      Note:
      sync automatically uses concurrent upload/download for large objects.
      If an object is larger than --part-size, COSCLI will split the object into multiple parts according to --part-size and use --thread-num threads to concurrently upload/download the object.
      Each thread maintains a URL. For each URL, you can use the --rate-limiting parameter to limit the speed of a single URL. When concurrent upload/download is enabled, the total rate is --thread-num * --rate-limiting.
      If an object is uploaded/downloaded in parts, checkpoint restart will be enabled by default.
      --include and --exclude support standard regular expression syntax, so you can use them to filter out objects that meet specific criteria.
      When using zsh, you may need to add double quotes at both ends of the pattern string.
      Do not set snapshot-path to the directory to be migrated or its subdirectories. 
        ./coscli sync ~/test/ cos://bucket1/example/ -r --include ".*.txt" --snapshot-path=/path/snapshot-path  --meta=x-cos-meta-a:a#ContentType:text#Expires:2022-10-12T00:00:00.000Z
      When you use the sync command, in addition to PUT request fees, there are two other scenarios where you will have a HEAD request for cloud files, which will incur additional fees:
      If the parameter to specify the snapshot directory (`--snapshot-path`) is not added, HEAD request fees will be incurred.
      If the parameter to specify the snapshot directory (`--snapshot-path`) is added and it is the first time the snapshot directory is generated, HEAD request fees will be incurred. If it is not the first time the snapshot directory is generated, no additional request fees will be incurred.
      For other common options of this command (such as switching bucket and user account), see Common Options.

      Examples

      Syncing object upload

      ./coscli sync ~/example.txt cos://bucket1/example.txt

      Syncing object download

      ./coscli sync cos://bucket1/example.txt ~/example.txt

      Syncing intra-bucket replication

      ./coscli sync cos://bucket1/example.txt cos://bucket1/example_copy.txt

      Syncing cross-bucket replication

      ./coscli sync cos://bucket1/example.txt cos://bucket2/example_copy.txt
      
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy