Check Filesystems on a Terminal that is Stuck During the Boot Process Without Shell Access
Objectives
1. Determine if this article is the correct choice for the
present issue.
2. Determine if a filesystem check is necessary.
3. Determine which filesystem needs to be checked.
4. Determine which OS image version is installed on the terminal.
5. Enter into a dracut shell.
6. Run appropriate commands to check filesystem(s).
7. Check for orphaned files.
Prerequisites
1. Physical access to the terminal.
2. USB keyboard.
3. Picture of terminal from customer.
Instructions
Determine if this article is the correct choice for the present issue
This article is only for terminals that hang during the boot process and never drop the user into a shell. If the user has shell access and can type commands, please refer to https://kb.fullcount.net/howto/check-filesystems-terminal.
To verify that this article applies to the current situation, ask the user for a picture of the display and one or more of the following messages will be displayed somewhere on the screen:
Determine if a filesystem check is necessary
The filesystem(s) on a terminal may become corrupt and need to be checked in the following scenarios:
If the message "[Failed] Failed to start File System Check on ______" appears on the screen during boot-up, then a filesystem check is necessary.
Determine which filesystem needs to be checked
The names of filesystems vary between OS image versions. The messages displayed on the screen will indicate which filesystem needs to be checked first; however, they will not tell you if more than one filesystem needs to be checked. To avoid repeating this whole process (i.e. partial boot, log in, check one filesystem, reboot) multiple times, it is recommended that you check all of the filesystems on the terminal before rebooting.
Determine which OS image version is installed on the terminal
The OS image version can usually be found by using the Device Management application: https://internal.fullcount.net/device-management/login.xhtml
1. Login to Device Management.
2. Click on the "Net Settings" tab.
3. Filter by "Serial Number" or other relevant filter. Enter
information in text field to search for.
4. Click on the "Find" button.
5. Use the scrollbar to scroll to the right.
6. The OS Version can be found below the "OS Version" field.
Enter into a dracut shell
Since we do not have normal shell access, we will have to manually load a dracut shell. This is done by appending kernel parameters to GRUB. Grub is the first menu that shows a FullCount logo and is what loads directly after the power-on self test (POST). It includes a countdown timer with a green selection bar.
1. Attach a keyboard and turn on the device.
2. When GRUB loads press "E" on the keyboard. A black box will appear
with text. On 7.3.0+ the GRUB username is fc_user and the password is @11ianc3.
3. Press the down arrow until a line appears that begins with either
"linux" or "linuxefi." This will be approximately 15 lines
down.
4. With the cursor on the same line as "linux" or
"linuxefi" press the "End" key on the keyboard. This will
move the cursor to the end of the line and will appear as though the cursor is
on a differnt line.
5. Ensure there's a space between the last entry on the line and the cursor.
Add 'rd.shell rd.debug' without quotes and with the space to the end of the
line.
6. Press "F10" to boot, and if successful, shell access should be
available after a brief moment.
Run appropriate commands to check filesystem(s)
|
OS Image Version |
Command(s) to Check Filesystem(s) |
|
6.0.X |
fsck -y /dev/mapper/systemvg-LVRoot |
|
6.1.X |
fsck -y /dev/sda2 fsck -y /dev/mapper/systemvg-LVRoot |
|
6.2.X and greater |
fsck -y /dev/sda2 fsck -y /dev/mapper/systemvg-LVRoot |
In order to reboot from the command line, type "exit" to leave the shell, and then press "Ctrl+Alt+Del" to reboot.
Check for orphaned files
Every time you run a filesystem check, there is a risk that data stored in the filesystem being checked will be lost. This is not a bug in the filesystem or OS, and there is no fix or work-around. After running a filesystem check, you need to check for orphaned (i.e. lost) files on each filesystem. In the top level directory of each filesystem (i.e. "/", "/usr", "/var", etc.), there is a directory named "lost+found". The "lost+found" directories contain files that were partially recovered by the filesystem check. These directories should be empty; if they aren't, the filesystem check was not able to completely repair the filesystem. At that point, your options are to manually examine each file and determine where it goes, or reimage the terminal.
Use the 'ls' command to obtain a list of files in a directory. For example, to display all of the files in the "/lost+found" directory, run the following command:
|
OS Image Version |
Orphan File Directories |
|
6.0.x |
/lost+found |
|
6.1.x |
/lost+found |
|
6.2.x and greater |
/lost+found |
Tips, Tricks and Reminders
1. You CANNOT run a filesystem check on a mounted
filesystem. Running a filesystem check on a mounted filesystem will completely
destroy the filesystem and you will have to re-image the machine. The fsck
command will warn you if you try to run a filesystem check on a mounted
filesystem, and ask if you want to continue; always choose "n" for
no.
2. Do not type "yes" at the "(repair filesystem) #" command
prompt. This is not a question, it is just the command prompt.
3. Use ↑ (up arrow) on the keyboard to browse through the shell command
history. You can save a lot of time by editing similar commands instead of
typing them in from scratch.