Using the rescue mode to troubleshoot Linux system startup failures
Error description
The instance cannot be started up normally after the Linux system restarts. There are many FAILED
startup failure items in the prompt message.
Possible cause
The key system files such as. .bin
and .lib
files are missing.
Solutions
See Steps to enter the instance rescue mode through the console to troubleshoot. Steps
1. Before you enter the rescue mode, we strongly recommend you back up the instance to avoid the impact of maloperations. You can create snapshots to back up cloud disks and create a custom image to back up local system disks. 2. Log in to the CVM console. On the "Instances" page, find the target instance, select More > Ops and check > Enter rescue mode.
3. In the pop-up window, set the instance login password for the rescue mode.
4. Click Enter rescue mode, and the instance status will change to "Entering rescue mode", which typically completes within a few minutes:
The status of instance entered the rescue mode changes to "Rescue mode" with a red exclamation mark.
5. Use the root
account and the password set in step 3 to log in to the instance as follows: 6. This document takes login via VNC as an example. After successful login, run the following commands in sequence to mount the root partition of the system disk.
Note:
In rescue mode, the device name of the instance system disk is vda
, and its root partition is vda1
, which is unmounted by default.
The returned result is shown below:
7. After successful mounting, you can manipulate the data in the root partition of the original system.
You can also use the mount -o bind
command to mount some sub-directories in the original file system and use the chroot
command to run commands in the specified root directory. Below are the specific commands:
mount -o bind /dev /mnt/vm1/dev
mount -o bind /dev/pts /mnt/vm1/dev/pts
mount -o bind /proc /mnt/vm1/proc
mount -o bind /run /mnt/vm1/run
mount -o bind /sys /mnt/vm1/sys
chroot /mnt/vm1 /bin/bash
When running the chroot
command:
If there is no error message, you can continue to run the cd /
command.
If the error message as shown below appears, the root directory cannot be switched normally. In this case, you can run cd /mnt/vm1
to view the root partition data.
8. Through the command, you can check that all files in the /usr/bin
directory in the original system root partition have been deleted.
9. In this case, you can create a normal instance using the same operating system, and run the following commands to compress and remotely copy the files in the /usr/bin
directory of the normal system to the abnormal instance.
For the normal instance, run the following commands in sequence:
cd /usr/bin/ && tar -zcvf bin.tar.gz *
scp bin.tar.gz root@abnormal instance ip:/mnt/vm1/usr/bin/
Note:
If the instances have public network IPs, the copy can be performed through the public network; otherwise, the copy is performed through the private network.
The execution result is as shown below:
For the abnormal instance, run the following commands in sequence in the rescue mode:
chroot /mnt/vm1 /bin/bash
The execution result is as shown below:
10. After repairing the instance, select More > Ops and check > Exit rescue mode under the Operation column of the target instance.
11. After exiting the rescue mode, the instance is in a shutdown status. Start up the instance to verify the system. As shown below, the system has been restored.
Was this page helpful?