tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
Cloud Object Storage
    Documentation

    COSFS

    Last updated: 2024-12-03 17:27:11

      FAQs About Features

      How do I mount a bucket with a temporary key?

      You can mount a bucket using a temporary key (STS) by following the steps below:
      Step 1. Create a temporary key configuration file, such as /tmp/passwd-sts, for the COSFS command option "-opasswd-file=[path]" to specify the key configuration file. For more information on temporary keys, see Generating and Using Temporary Keys. Below is an example of a temporary key configuration file:
      COSAccessKeyId=AKIDXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  # Below are the Id, Key, and Token fields of the temporary key
      COSSecretKey=GYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
      COSAccessToken=109dbb14ca0c30ef4b7e2fc9612f26788cadbfac3
      COSAccessTokenExpire=2017-08-29T20:30:00  # Expiration time of the temporary token in GMT time, which should be in the same format as shown here
      COSFS will determine whether it needs to reload the configuration from the key file based on the time configured in COSAccessTokenExpire.
      Note:
      To prevent key leakage, you should set the permission of the key file to 600 in COSFS by running the following command:
      chmod 600 /tmp/passwd-sts
      Step 2: Run the COSFS command. Specify the role as sts using the command option -ocam_role=[role], and specify the key file's path using -opasswd_file=[path], as shown below:
      cosfs examplebucket-1250000000 /mnt/cosfs -ourl=http://cos.ap-guangzhou.myqcloud.com -odbglevel=info -oallow_other -ocam_role=sts -opasswd_file=/tmp/passwd-sts

      How do I check the mounting parameter options and version number provided by COSFS?

      You can check the parameter options provided by COSFS using the cosfs --help command, and check the COSFS version number using cosfs --version.

      How do I view the logs generated by COSFS?

      In CentOS, COSFS-generated logs are stored in /var/log/messages; in Ubuntu, these logs are stored in /var/log/syslog. If you have any problems during operation, send us the log for the corresponding time period.

      How do I mount a directory in a bucket?

      When running the mounting command, you can specify a directory under in bucket as shown below:
      cosfs examplebucket-1250000000:/my-dir /mnt/cosfs -ourl=http://cos.ap-guangzhou.myqcloud.com -odbglevel=info
      Note:
      my-dir must begin with /.
      In versions below v1.0.5, the mounting command is:
      cosfs 1250000000:examplebucket:/my-dir /mnt/cosfs -ourl=http://cos.ap-guangzhou.myqcloud.com -odbglevel=info

      How does a non-root user mount COSFS?

      If you are not a root user, you are recommended to create a .passwd-cosfs file in the Home directory, set the permission to 600, and then mount COSFS using the mounting command. In addition, you can also specify the key file path using the -opasswd_file=path option and set the permission to 600.

      Does COSFS support mounting over HTTPS?

      Yes. Mounting methods over HTTP and HTTPS are as shown below:
      -ourl=http://cos.ap-guangzhou.myqcloud.com
      -ourl=https://cos.ap-guangzhou.myqcloud.com
      If the version of the NSS library that libcurl depends on is v3.12.3 or higher (check the NSS version using curl -V), you can mount the bucket over HTTPS by running the following command:
      echo "export NSS_STRICT_NOFORK=DISABLED" >> ~/.bashrc
      source ~/.bashrc

      How do I set auto-mounting at startup for COSFS?

      You need to install the FUSE package first:
      # CentOS
      #sudo yum install -y fuse
      
      # Ubuntu
      #sudo apt-get install fuse
      Add the following to the /etc/fstab file. The _netdev option specifies that the command can be executed only after the network is ready:
      cosfs#examplebucket-1250000000 /mnt/cosfs fuse _netdev,allow_other,url=http://cos.ap-guangzhou.myqcloud.com,dbglevel=info

      How do I set the user and user group of files or directories under a mount point?

      In certain scenarios (such as NGINX server), you need to set the user and user group of files or directories under a mount point, such as user www (uid = 1002, gid =1002). In this case, you need to add the following mount parameters:
      -ouid=1002 -ogid=1002

      How do I mount multiple buckets?

      If you need to mount multiple buckets simultaneously, write a line for each bucket to be mounted in the /etc/passwd-cosfs configuration file, with the content of each line being in the same format as that for a single bucket, such as:
      echo examplebucket-1250000000:AKIDXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:GYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY >> /etc/passwd-cosfs
      echo log-1250000000:AKIDXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:GYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY >> /etc/passwd-cosfs

      How do I make a mounted directory accessible to other accounts on the server?

      To make the mounted directory accessible to other accounts, specify -oallow_other during mounting.

      Are there any restrictions on filenames in a COSFS-mounted directory?

      In a COSFS-mounted directory, you can create a file whose name does not contain /. On a Unix-like system, the / character is a directory separator, so / is not allowed in filenames in a COSFS-mounted directory. In addition, when creating a file whose name contains special characters, make sure that such characters are not used by shell; otherwise, file creation may fail.

      How does COSFS determine whether a file exists?

      In the internal logic of COSFS, a HEAD request is used to determine whether the parent directory and the file exist.

      How can I view the storage usage with COSFS?

      COSFS does not support viewing the storage usage. If you need statistics on bucket usage and your data volume is small, log in to the COS console to view the usage. If the data volume is high, you can leverage the Inventory feature.

      How do I check which processes have accessed the mounted directory?

      Run the following command to check which processes have accessed the mounted directory, such as /mnt/cosfs.
      lsof /mnt/cosfs

      FAQs About Troubleshooting

      What should I do if the error message "unable to access MOUNTPOINT /path/to/mountpoint: Transport endpoint is not connected" is displayed and COSFS becomes inaccessible?

      You can check whether the COSFS process exists using the ps ax|grep cosfs command. If the COSFS process is down due to faulty operations, run the following command to remount it:
      umount -l /path/to/mnt_dir
      cosfs examplebucket-1250000000:/my-dir /mnt/cosfs -ourl=http://cos.ap-guangzhou.myqcloud.com -odbglevel=info
      If the COSFS process error is not caused by faulty operations, you can check whether the fuse version of the server is below v2.9.4. The libfuse on versions below v2.9.4 can cause exceptional exit of the COSFS process. In this case, it is recommended to update the fuse version or install the latest version of COSFS as described in COSFS.

      What should I do if the Content-Type of a file uploaded via COSFS is changed to "application/octet-stream"?

      COSFS automatically sets the Content-Type of a file uploaded to COS by comparing the file's suffix against /etc/mime.types. When a Content-Type problem occurs, check whether the configuration file exists on the system. For Ubuntu, you can add it using sudo apt-get install mime-support. For CentOS, you can add it using sudo yum install mailcap. You can also create this file manually, where one line is added for each file format, as shown below:
      image/jpeg                                      jpg jpeg
      image/jpm                                       jpm jpgm
      image/jpx                                       jpx jpf

      What should I do if "Bucket not exist" is displayed during mounting?

      Check the parameter -ourl to ensure that the bucket part is not included in the URL. The correct format is:
      -ourl=http://cos.ap-guangzhou.myqcloud.com

      Why can't I write to files that I could write to previously?

      Because of the adjustment to the COS authentication policy, using the COSFS tool below v1.0.0 will cause the policy check to fail. You can install the latest COSFS tool and mount it again.

      What should I do if an error such as "Input/Output ERROR" occurs when using the COSFS tool?

      Please follow the steps below to identify the cause of the error:
      1. Check whether the server can access the COS domain name normally.
      2. Check whether the account is configured correctly.
      3. If you used the cp command containing the -p or -a parameter for copying, it is recommended that you remove the parameter and run the command again.
      After confirming that the above configurations are correct, open the /var/log/messages log file on the server and locate the log entry for s3fs, which can help you identify the cause of error. If the error persists, contact us.

      I've set auto-amounting at startup for COSFS using /etc/fstab, but the error "wrong fs type, bad option, bad superblock on cosfs" occurs when I run "mount -a". Why?

      This error happens generally because the fuse library is missing on your server. It is recommended to install the fuse library by running the following command:
      CentOS
      sudo yum install fuse
      Ubuntu
      sudo apt-get install fuse

      Is it normal to see a lot of 404 error codes in the system log /var/log/messages?

      In the internal logic of COSFS, a HEAD request is used to determine whether the parent directory and the file exist, and the 404 error does not necessarily mean that the program is running erroneously.

      Why does the file that I see in COS have a size of 0?

      Generally, when you write data to COSFS, a file with a size of 0 will be first created in COS, and the data will be written to the local cache file. During the write process, the result of the mount point ls command shows the change in file size. When the file is closed, COSFS will upload the data written to the local cache file to COS. If the upload fails, you may only see a file with a size of 0, in which case you can try copying the failed file again.

      Are the files in the COSFS cache directory the same as those in COS? Can I use them directly?

      No. The files in the cache directory are used to accelerate reads and writes in COSFS and may include only a part of the files in COS.

      I have copied a file to COSFS using the rsync command and the progress has reached 100%, but I only see a temporary file on the server. Why?

      The rsync command creates a temporary file in the mounted directory. The 100% progress only means that the temporary file has been 100% written locally. After that, it will be uploaded to COS and then renamed and copied. Typically, it takes more time to copy data to a COSFS-mounted directory using the rsync command than using the cp command.

      What should I do if COSFS uses up the disk capacity?

      Uploads and downloads using COSFS involve disk file caching. When you upload or download a large file, if you don't specify the -oensure_diskfree=[size] parameter, the disk where the file is cached will be used up. You can use the -oensure_diskfree parameter to specify that when the remaining capacity of the disk is less than [size] MB, COSFS will minimize disk capacity usage (in MB). If you specify the -ouse_cache=[path] parameter, the cache file will be located in the path directory; otherwise, it will be in the /tmp directory.
      For example, you can configure COSFS to reduce disk capacity usage when the remaining capacity is less than 10 GB by running the following command:
      cosfs examplebucket-1250000000 /mnt/cosfs -ourl=http://cos.ap-guangzhou.myqcloud.com -odbglevel=info -oensure_diskfree=10240

      What should I do if the error message "fuse: failed to open /dev/fuse: Operation not permitted" is displayed when I use docker to mount COSFS?

      To start a docker image, you need to add the parameter --privileged.

      Can I use a directory as a shared cache directory for multiple mount points?

      It is not recommended that multiple mount points share a cache directory. The cache directory contains metadata used by COSFS, and sharing it may mess up the metadata.

      What should I do if the error message "/bin/mount:unrecognized option --no-canonicalize" is displayed when I use COSFS for mounting?

      Lower versions of the mounting tool do not support the --no-canonicalize option. Please update the tool. You are recommended to download v2.17 and then mount it again. The installation command is as follows:
      tar -jxvf util-linux-ng-2.17.tar.bz2
      cd util-linux-ng-2.17
      ./configure --prefix=/usr/local/util-linux-ng
      make && make install
      mv /bin/mount{,.off}
      ln -s /usr/local/util-linux-ng/bin/mount /bin

      What should I do if mounting fails?

      Step 1. Check whether the mounting command and the key configuration file are correct and whether the COS service is accessible based on the log files and error messages.
      Step 2. Check whether the server time is correct by running the date command. If it is incorrect, correct it using the date command and then mount it again, such as date -s '2014-12-25 12:34:56'.

      Is it normal that the time of a mounted directory changes to 1970-01-01 08:00 when I use the ls -l --time-style=long-iso command?

      Yes. After you unmount the mount point, the time of the mounted directory will return back to the time before mounting.

      Can a mounted directory be non-empty?

      You can mount a non-empty directory using the -ononempty parameter, but you are not recommended to do so because a problem may occur when the mount point and the original directory have files with the same path.

      Why does it take the ls command so long to return when I run it in a COSFS directory?

      If there are a lot of files in a mounted directory, executing the Is command requires a HEAD operation on each file in the directory, so it takes a lot of time to read the directory system before the command returns.
      Note:
      You are recommended not to enable IO hung which may result in unnecessary restarts.

      What should I do if the system log files generated take up a lot of storage space when I set the log level to info?

      You can periodically clear the generated system log files or increase the log level. For example, you can use '-odbglevel=crit' for mounting.

      What scenarios does COSFS apply to, and how about the read and write performance?

      As disks are required for COSFS′s read and write operations, COSFS is only suitable for scenarios where POSIX access syntax is required for COS access, such as machine learning algorithms of shared data sets reading shared data and simple log backups. COSFS adopts multithreaded upload and download for acceleration. Via a private network in the same region, it takes COSFS about 80s to sequentially read a 6 GB file and about 160s to sequentially write a 6 GB file. Generally, you can use SDK and multi-thread to achieve better performance.
      Note:
      The large number of system calls caused by file reading and writing, along with the large number of logs, can affect the COSFS read and write performance to some extent. If you have high performance requirements, you can specify -odbglevel=warn or higher log level.

      What should I do if the system prompts that COSFS is not found after the COSFS RPM package is installed?

      The installation path of COSFS is /usr/local/bin. If the system prompts that COSFS is not found, the possible cause is that the path is not specified in the PATH environment variable, and you need to add a line of configuration to ~/.bashrc:
      export PATH=/usr/local/bin:$PATH
      Then you need to run the following command:
      source ~/.bashrc

      What should I do if the system reports "conflicts with file from package fuse-libs-*" when I install the COSFS RPM package?

      Add the --force option when installing the COSFS RPM package:
      rpm -ivh cosfs-1.0.19-centos7.0.x86_64.rpm --force

      Why does the system indicate no permission when I separately mount a directory that has been granted read-only permission in COSFS?

      COSFS requires the GetBucket permission on the root directory. Therefore, you need to add the GetBucket permission on the root directory and the read permission on the corresponding directory. In that way, you can list other directories but have no operation permission on them.

      What should I do if COSFS has a high CPU utilization, sends a large number of HEAD and LIST requests to COS, and incurs a large amount of request fees during a certain period of time every day?

      This is usually caused by scheduled disk scanning tasks on your server. A common disk scanning program on Linux is updatedb. You can add the COSFS-mounted directory to the PRUNEPATHS configuration item in updatedb's configuration file /etc/updatedb.conf to prevent disk scanning. In addition, you can use the Linux tool auditd to find programs that access the COSFS mount point.
      Follow the steps below:
      Step 1: Install auditd.
      Ubuntu:
      ap-get install auditd -y
      CentOS:
       yum install audit audit-libs
      Step 2: Start the auditd service.
      systemctl start auditd
      systemctl enable auditd
      Step 3: Monitor the mounted directory.
      Note:
      -w specifies the COSFS-mounted directory, and -k indicates the key to be outputted to the audit logs.
      auditctl -w /usr/local/service/mnt/ -k cosfs_mnt
      Step 4: Based on the logs, determine the programs that access the COSFS mount point.
      The audit logs are in the /var/log/audit directory and can be queried by running the following command:
      ausearch -i|grep 'cosfs_mnt'
      Step 5: Stop the auditd service. You can run the following command to stop the auditd service:
      /sbin/service auditd stop
      Note:
      If the program accessing the mount point is always running, the newly started auditd will not monitor the access behavior of the program. This is because only the first call in multiple calls from the program to the mounted directory will be recorded.

      Why are the values of Size and Available are 256 TB after I run df?

      COS buckets offer unlimited storage capacity. The 256 TB displayed is only used as the output of df.

      Why is the value of Used is 0 after I run df?

      COSFS does not occupy local storage. In order to be compatible with tools such as df, the values of Size, Used, and Available displayed in COSFS are not the actual values.

      Why are the values of Inode, IUsed, and IFree are 0 after I run df -i?

      COSFS is not a disk-based file system and thus does not have inode.
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy