You can mount a bucket using a temporary key (STS) by following the steps below:
Step 1. Create a temporary key configuration file, such as /tmp/passwd-sts, for the COSFS command option "-opasswd-file=[path]" to specify the key configuration file. For more information on temporary keys, see Generating and Using Temporary Keys. Below is an example of a temporary key configuration file:
COSAccessKeyId=AKIDXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # Below are the Id, Key, and Token fields of the temporary key
COSSecretKey=GYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
COSAccessToken=109dbb14ca0c30ef4b7e2fc9612f26788cadbfac3
COSAccessTokenExpire=2017-08-29T20:30:00 # Expiration time of the temporary token in GMT time, which should be in the same format as shown here
COSFS will determine whether it needs to reload the configuration from the key file based on the time configured in COSAccessTokenExpire.
Note:To prevent key leakage, you should set the permission of the key file to 600 in COSFS by running the following command:
chmod 600 /tmp/passwd-sts
Step 2: Run the COSFS command. Specify the role as sts using the command option -ocam_role=[role]
, and specify the key file's path using -opasswd_file=[path]
, as shown below:
cosfs examplebucket-1250000000 /mnt/cosfs -ourl=http://cos.ap-guangzhou.myqcloud.com -odbglevel=info -oallow_other -ocam_role=sts -opasswd_file=/tmp/passwd-sts
You can check the parameter options provided by COSFS using the cosfs --help
command, and check the COSFS version number using cosfs --version
.
In CentOS, COSFS-generated logs are stored in /var/log/messages
; in Ubuntu, these logs are stored in /var/log/syslog
. If you have any problems during operation, send us the log for the corresponding time period.
When running the mounting command, you can specify a directory under in bucket as shown below:
cosfs examplebucket-1250000000:/my-dir /mnt/cosfs -ourl=http://cos.ap-guangzhou.myqcloud.com -odbglevel=info
Note:my-dir must begin with
/
.
In versions below v1.0.5, the mounting command is:
cosfs 1250000000:examplebucket:/my-dir /mnt/cosfs -ourl=http://cos.ap-guangzhou.myqcloud.com -odbglevel=info
If you are not a root user, you are recommended to create a .passwd-cosfs file in the Home
directory, set the permission to 600, and then mount COSFS using the mounting command. In addition, you can also specify the key file path using the -opasswd_file=path
option and set the permission to 600.
Yes. Mounting methods over HTTP and HTTPS are as shown below:
-ourl=http://cos.ap-guangzhou.myqcloud.com
-ourl=https://cos.ap-guangzhou.myqcloud.com
If the version of the NSS library that libcurl depends on is v3.12.3 or higher (check the NSS version using curl -V
), you can mount the bucket over HTTPS by running the following command:
echo "export NSS_STRICT_NOFORK=DISABLED" >> ~/.bashrc
source ~/.bashrc
You need to install the FUSE package first:
# CentOS
#sudo yum install -y fuse
# Ubuntu
#sudo apt-get install fuse
Add the following to the /etc/fstab
file. The _netdev
option specifies that the command can be executed only after the network is ready:
cosfs#examplebucket-1250000000 /mnt/cosfs fuse _netdev,allow_other,url=http://cos.ap-guangzhou.myqcloud.com,dbglevel=info
In certain scenarios (such as NGINX server), you need to set the user and user group of files or directories under a mount point, such as user www
(uid = 1002, gid =1002). In this case, you need to add the following mount parameters:
-ouid=1002 -ogid=1002
If you need to mount multiple buckets simultaneously, write a line for each bucket to be mounted in the /etc/passwd-cosfs
configuration file, with the content of each line being in the same format as that for a single bucket, such as:
echo examplebucket-1250000000:AKIDXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:GYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY >> /etc/passwd-cosfs
echo log-1250000000:AKIDXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:GYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY >> /etc/passwd-cosfs
To make the mounted directory accessible to other accounts, specify -oallow_other
during mounting.
In a COSFS-mounted directory, you can create a file whose name does not contain /
. On a Unix-like system, the /
character is a directory separator, so /
is not allowed in filenames in a COSFS-mounted directory. In addition, when creating a file whose name contains special characters, make sure that such characters are not used by shell; otherwise, file creation may fail.
In the internal logic of COSFS, a HEAD request is used to determine whether the parent directory and the file exist.
COSFS does not support viewing the storage usage. If you need statistics on bucket usage and your data volume is small, log in to the COS console to view the usage. If the data volume is high, you can leverage the Inventory feature.
Run the following command to check which processes have accessed the mounted directory, such as /mnt/cosfs
.
lsof /mnt/cosfs
You can check whether the COSFS process exists using the ps ax|grep cosfs
command. If the COSFS process is down due to faulty operations, run the following command to remount it:
umount -l /path/to/mnt_dir
cosfs examplebucket-1250000000:/my-dir /mnt/cosfs -ourl=http://cos.ap-guangzhou.myqcloud.com -odbglevel=info
If the COSFS process error is not caused by faulty operations, you can check whether the fuse version of the server is below v2.9.4. The libfuse on versions below v2.9.4 can cause exceptional exit of the COSFS process. In this case, it is recommended to update the fuse version or install the latest version of COSFS as described in COSFS.
COSFS automatically sets the Content-Type of a file uploaded to COS by comparing the file's suffix against /etc/mime.types. When a Content-Type problem occurs, check whether the configuration file exists on the system. For Ubuntu, you can add it using sudo apt-get install mime-support
. For CentOS, you can add it using sudo yum install mailcap
. You can also create this file manually, where one line is added for each file format, as shown below:
image/jpeg jpg jpeg
image/jpm jpm jpgm
image/jpx jpx jpf
Check the parameter -ourl
to ensure that the bucket part is not included in the URL. The correct format is:
-ourl=http://cos.ap-guangzhou.myqcloud.com
Because of the adjustment to the COS authentication policy, using the COSFS tool below v1.0.0 will cause the policy check to fail. You can install the latest COSFS tool and mount it again.
Please follow the steps below to identify the cause of the error:
cp
command containing the -p
or -a
parameter for copying, it is recommended that you remove the parameter and run the command again.After confirming that the above configurations are correct, open the /var/log/messages
log file on the server and locate the log entry for s3fs, which can help you identify the cause of error. If the error persists, contact us.
This error happens generally because the fuse library is missing on your server. It is recommended to install the fuse library by running the following command:
CentOS
sudo yum install fuse
Ubuntu
sudo apt-get install fuse
In the internal logic of COSFS, a HEAD request is used to determine whether the parent directory and the file exist, and the 404 error does not necessarily mean that the program is running erroneously.
Generally, when you write data to COSFS, a file with a size of 0 will be first created in COS, and the data will be written to the local cache file.
During the write process, the result of the mount point ls command shows the change in file size. When the file is closed, COSFS will upload the data written to the local cache file to COS. If the upload fails, you may only see a file with a size of 0, in which case you can try copying the failed file again.
No. The files in the cache directory are used to accelerate reads and writes in COSFS and may include only a part of the files in COS.
The rsync command creates a temporary file in the mounted directory. The 100% progress only means that the temporary file has been 100% written locally. After that, it will be uploaded to COS and then renamed and copied.
Typically, it takes more time to copy data to a COSFS-mounted directory using the rsync command than using the cp command.
Uploads and downloads using COSFS involve disk file caching. When you upload or download a large file, if you don't specify the -oensure_diskfree=[size] parameter, the disk where the file is cached will be used up. You can use the -oensure_diskfree parameter to specify that when the remaining capacity of the disk is less than [size] MB, COSFS will minimize disk capacity usage (in MB). If you specify the -ouse_cache=[path] parameter, the cache file will be located in the path directory; otherwise, it will be in the /tmp directory.
For example, you can configure COSFS to reduce disk capacity usage when the remaining capacity is less than 10 GB by running the following command:
cosfs examplebucket-1250000000 /mnt/cosfs -ourl=http://cos.ap-guangzhou.myqcloud.com -odbglevel=info -oensure_diskfree=10240
To start a docker image, you need to add the parameter --privileged.
It is not recommended that multiple mount points share a cache directory. The cache directory contains metadata used by COSFS, and sharing it may mess up the metadata.
Lower versions of the mounting tool do not support the --no-canonicalize
option. Please update the tool. You are recommended to download v2.17 and then mount it again. The installation command is as follows:
tar -jxvf util-linux-ng-2.17.tar.bz2
cd util-linux-ng-2.17
./configure --prefix=/usr/local/util-linux-ng
make && make install
mv /bin/mount{,.off}
ln -s /usr/local/util-linux-ng/bin/mount /bin
Step 1. Check whether the mounting command and the key configuration file are correct and whether the COS service is accessible based on the log files and error messages.
Step 2. Check whether the server time is correct by running the date command. If it is incorrect, correct it using the date command and then mount it again, such as date -s '2014-12-25 12:34:56'
.
ls -l --time-style=long-iso
command?Yes. After you unmount the mount point, the time of the mounted directory will return back to the time before mounting.
You can mount a non-empty directory using the -ononempty
parameter, but you are not recommended to do so because a problem may occur when the mount point and the original directory have files with the same path.
ls
command so long to return when I run it in a COSFS directory?If there are a lot of files in a mounted directory, executing the Is
command requires a HEAD operation on each file in the directory, so it takes a lot of time to read the directory system before the command returns.
Note:You are recommended not to enable IO hung which may result in unnecessary restarts.
You can periodically clear the generated system log files or increase the log level. For example, you can use '-odbglevel=crit' for mounting.
As disks are required for COSFS′s read and write operations, COSFS is only suitable for scenarios where POSIX access syntax is required for COS access, such as machine learning algorithms of shared data sets reading shared data and simple log backups. COSFS adopts multithreaded upload and download for acceleration. Via a private network in the same region, it takes COSFS about 80s to sequentially read a 6 GB file and about 160s to sequentially write a 6 GB file. Generally, you can use SDK and multi-thread to achieve better performance.
Note:The large number of system calls caused by file reading and writing, along with the large number of logs, can affect the COSFS read and write performance to some extent. If you have high performance requirements, you can specify
-odbglevel=warn
or higher log level.
The installation path of COSFS is /usr/local/bin
. If the system prompts that COSFS is not found, the possible cause is that the path is not specified in the PATH
environment variable, and you need to add a line of configuration to ~/.bashrc
:
export PATH=/usr/local/bin:$PATH
Then you need to run the following command:
source ~/.bashrc
Add the --force
option when installing the COSFS RPM package:
rpm -ivh cosfs-1.0.19-centos7.0.x86_64.rpm --force
COSFS requires the GetBucket permission on the root directory. Therefore, you need to add the GetBucket permission on the root directory and the read permission on the corresponding directory. In that way, you can list other directories but have no operation permission on them.
This is usually caused by scheduled disk scanning tasks on your server. A common disk scanning program on Linux is updatedb. You can add the COSFS-mounted directory to the PRUNEPATHS
configuration item in updatedb's configuration file /etc/updatedb.conf
to prevent disk scanning. In addition, you can use the Linux tool auditd to find programs that access the COSFS mount point.
Follow the steps below:
Step 1: Install auditd.
Ubuntu:
ap-get install auditd -y
CentOS:
yum install audit audit-libs
Step 2: Start the auditd service.
systemctl start auditd
systemctl enable auditd
Step 3: Monitor the mounted directory.
Note:
-w
specifies the COSFS-mounted directory, and-k
indicates the key to be outputted to the audit logs.
auditctl -w /usr/local/service/mnt/ -k cosfs_mnt
Step 4: Based on the logs, determine the programs that access the COSFS mount point.
The audit logs are in the /var/log/audit directory and can be queried by running the following command:
ausearch -i|grep 'cosfs_mnt'
Step 5: Stop the auditd service.
You can run the following command to stop the auditd service:
/sbin/service auditd stop
Note:If the program accessing the mount point is always running, the newly started auditd will not monitor the access behavior of the program. This is because only the first call in multiple calls from the program to the mounted directory will be recorded.
Size
and Available
are 256 TB after I run df
?COS buckets offer unlimited storage capacity. The 256 TB displayed is only used as the output of df
.
Used
is 0 after I run df
?COSFS does not occupy local storage. In order to be compatible with tools such as df
, the values of Size
, Used
, and Available
displayed in COSFS are not the actual values.
Inode
, IUsed,
and IFree
are 0 after I run df -i
?COSFS is not a disk-based file system and thus does not have inode
.
Was this page helpful?