Diagnose and Replace a Defective Hard Drive (Linux Dedicated Server with Software RAID)
In this article, you will learn how to identify a defective hard drive on a Linux Dedicated Server and how to prepare for the replacement.
Please Note
This article assumes that you have basic knowledge of Linux server administration. If you have any questions or need assistance regarding the replacement of a defective hard drive, please contact IONOS Customer Support.
In order to ensure the highest possible reliability, it is necessary that you monitor the software RAID of your Dedicated Servers. If you discover that a hard drive is defective or receive a notification email about a defective hard drive, you must contact customer service to arrange for the hard drive replacement. This requires that you identify the defective hard drive and prepare the server for the replacement.
Please Note
RAID systems enable greater reliability and/or higher speed capabilities. However, they are not a substitute for regular backups. To avoid data loss, we recommend that you back up your data regularly. Also, make sure that you back up your data before performing the steps below to ensure the security of your data. For more information on creating backups, click here:
Back Up Linux Server Data
Checking the Status of the Software RAID
To check the status of the software RAID, enter the following command in the shell:
[root@host~]: cat /proc/mdstat
If both hard drives are present and properly mounted, the following message will be displayed:
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
read_ahead 1024 sectors
md2 : active raid1 sda3[1] sdb3[0]
262016 blocks [2/2] [UU]
md1 : active raid1 sda2[1] sdb2[0]
119684160 blocks [2/2] [UU]
md0 : active raid1 sda1[1] sdb1[0]
102208 blocks [2/2] [UU]
unused devices: <none>
The example above shows three multiple devices or logical drives (md0, md1, md2). For each of these logical drives, you specify which partitions they are composed of and on which drives these partitions are located.
Example: The logical drive md0 consists of the partitions sda1 and sdb 1.
In the line listed below the logical drive, the state of the individual partitions is displayed in square brackets at the end of the line. A U means that the respective hard drive is integrated into the RAID (up).
In the following example, all logical drives have only one partition mounted which is located on the sda hard drive. The respective partition located on the second hard drive sdb is not included. You can also see this from the entry [_U]. The unmounted partitions of the hard drive sdb indicate that there is an error or a defect on this hard drive.
[root@localhost~]# cat /proc/mdstat
Personalities : [raid1]
read_ahead 1024 sectors
md0 : active raid1 sda1[1]
102208 blocks [2/1] [_U]
md1 : active raid1 sda2[1]
119684160 blocks [2/1] [_U]
md2 : active raid1 sda3[1]
262016 blocks [2/1] [_U]
unused devices: <none>
In the following example, a defective hard drive is still included in the RAID:
[root@localhost~]# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sda3[0] sdb3[2](F)
439553856 blocks super 1.0 [2/1] [U_]
bitmap: 1/4 pages [4KB], 65536KB chunk
md1 : active raid1 sdb1[2](F) sda1[0]
19529600 blocks super 1.0 [2/1] [U_]
unused devices: <none>
The entry (F) in this example shows that the partition is marked as faulty.
Diagnose Faults and Find Details Needed for a Hard Drive Replacement
In order to detect hard disk errors, we recommend that you follow these steps:
Install the Smartctl program. Smartctl is a command line program used to monitor volumes using SMART (Self-Monitoring, Analysis and Reporting Technology). With this program, you can check whether a hard drive is defective. It is a component of the Smartmontools. The Smartmontools are available as packages for many Linux distributions.
Please Note
A hard drive defect cannot always be detected by the smart values. Accordingly, we recommend that you always analyze the log file /var/log/messages.
Installing Smartctl
To install Smartctl, type the following command:
CentOS
yum install smartmontools
Ubuntu
sudo apt-get install smartmontools
Viewing Hard Drive Information
To view a list of hard drives, type the following command:
smartctl --scan
Example:
[root@8E8885C~] # smartctl --scan
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
To view detailed information for troubleshooting, type the following command:
smartctl -iHAl error [/dev/sd(a-z)]
Analyzing Results
Analyze the detailed information you received using the smartctl -iHAl error [NAMEOFTHEHARDDRIVE] command. The first section lists information that you can use to identify the hard drive:
=== START OF INFORMATION SECTION ===
Device Model: HGST HUS722T1TALA604
Serial Number: WMC6N0K2RW66
LU WWN Device Id: 5 0014ee 004722db0
Firmware Version: RAGNWA07
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri May 3 07:45:14 2019 UTC
SMART support is: Available - device has SMART capability.
SMART support is: EnabledThis section displays, among other things, the device model and serial number of the hard drive under test.
In the second section the current state of the hard drive is evaluated by Smartctl. If, for example, the value "Failed" or "UNKNOWN" is displayed instead of "PASSED", you should replace the hard drive as soon as possible.
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSEDIn the third section, the SMART VALUES determined are listed in detail. Next to each current percentage value (VALUE), the worst ever measured value (WORST) and the respective limit value (THRESH) are listed. If the current percentage value (VALUE) or the worst ever measured value (WORST) exceeds the limit value (THRESH), a SMART warning is displayed in the WHEN_FAILED column (e.g. FAILING_NOW).
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always 0
3 Spin_Up_Time 0x0027 183 183 021 Pre-fail Always 3833
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always 9
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always 0
9 Power_On_Hours 0x0032 097 097 000 Old_age Always 2560
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always 9
16 Unknown_Attribute 0x0022 000 200 000 Old_age Always 26802171994
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always 0
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always 4
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always 67
194 Temperature_Celsius 0x0022 116 111 000 Old_age Always 31
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline 0The following details can indicate an imminent hard drive failure before a SMART warning is displayed:
Reallocated_Sector_Ct: Specifies the number of sectors reassigned due to read errors. If a sector can no longer be read, written to or checked correctly, a replacement sector is automatically assigned to it. The faulty sector is permanently marked as unreadable. This is a clear warning sign for incipient surface problems. If this value is not equal to zero, a hard drive failure is often imminent. This value is the most important indicator for a hard drive replacement.
Current_Pending_Sector_Ct: Specifies the number of unstable sectors waiting for remapping. If a sector cannot be read and written correctly, it first receives the status Current Pending Sector. The sector is not reassigned in this state, since the data in the sector are unknown. Only after several unsuccessful read or write attempts is a replacement sector assigned and the faulty sector is permanently marked as unreadable. The value Current_Pending_Sector_Ct is an important indicator for a hard drive replacement. If this value is not equal to zero, a hard drive failure is often imminent.
Offline_Uncorrectable: Specifies the number of uncorrectable write and read sector errors.
The last section deals with the internal hard drive log. Errors are recorded here if the work orders of the servers have not been processed properly by the hard drive. If the number of errors in this section is at least two digits, you should replace the hard drive as soon as possible.
SMART Error Log Version: 1
No Errors LoggedCreating a SMART Log
To create a complete SMART log, type the following command:
smartctl -x [NAMEOFTHEHARDDRIVE]Example:
C:\Program Files\smartmontools\bin>smartctl -x /dev/sda -x
If the hard drive can no longer be accessed using Smartctl, you can pull up the required information using the hdparm program.
How to install hdparm:
CentOS
yum -y install hdparm
Ubuntu/Debian
sudo apt-get update
sudo apt-get install hdparm
Then, type the following command to access the information required for drive replacement:
hdparm -i /dev/sda
Please Note
If the SMART log was created as described above, it will contain enough sufficient information. You can use it to have the defective hard drive replaced. Please contact the customer service once you have it.
If you cannot find the serial number of the defective hard drive using Smartctl, you can alternatively provide customer service with the serial number of the working hard drive(s).
Preparing for Hard Drive Replacement
The following example assumes that the second hard drive (sdb) needs to be replaced. During the status check, for example, the following status of the software RAID is displayed:
[root@host~]# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sda3[0] sdb3[2]
439553856 blocks super 1.0 [2/1] [UU]
md1 : active raid1 sdb1[2] sda1[0]
19529600 blocks super 1.0 [2/1] [UU]
unused devices: <none>
In this example, the second hard drive (sdb) is still integrated in the RAID and is therefore still in operation.
Manually Mark Raid Device as "Faulty" to Remove it from the RAID
To mark the defective drive as faulty so that it can be removed from the RAID, enter the following command:
[root@host~]# mdadm PATHm_OFTHE_RAID_ARRAYS -f PATH_OFTHE_HARD_DRIVE
In the examples below, the hard drives sdb3 and sdb1 are marked as faulty:
[root@host~]# mdadm /dev/md3 -f /dev/sdb3
mdadm: set /dev/sdb3 faulty in /dev/md3
[root@host~]# mdadm /dev/md1 -f /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md1
After entering the command, the RAID has the following status:
[root@host~]# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sda3[0] sdb3[2](F)
439553856 blocks super 1.0 [2/1] [U_]
md1 : active raid1 sdb1[2](F) sda1[0]
19529600 blocks super 1.0 [2/1] [U_]
unused devices: <none>
Remove Partition/ from the Multiple Device
To remove a partition from the Multiple Device, type the following command:
[root@host~]# mdadm -r /PATH_OFTHE_RAID_ARRAYS /PATH_OFTHE_HARD_DRIVE
In the examples below, the hard drives sdb3 and sdb1 are removed from the multiple device md3 and md1 respectively:
[root@host~]# mdadm -r /dev/md3 /dev/sdb3
mdadm: hot removed /dev/sdb3 from /dev/md3
[root@host~]# mdadm -r /dev/md1 /dev/sdb1
mdadm: hot removed /dev/sdb1 from /dev/md1
Then check the status of the RAID. In this example, the RAID prepared for drive replacement has the following final state:
[root@host~]# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sda3[0]
439553856 blocks super 1.0 [2/1] [U_]
md1 : active raid1 sda1[0]
19529600 blocks super 1.0 [2/1] [U_]
unused devices: <none>
Checking the Swap Partitions Used
Check which swap partitions are used by the operating system. To do this, enter the following command:
[root@host~]# cat /proc/swaps
Filename Type Size Used Priority
/dev/sda2 partition 9765884 0 -1
/dev/sdb2 partition 9765884 0 -2
Alternatively, you can use the following command to check which swap partitions are defined in fstab:
[root@host~]# grep swap /etc/fstab
/dev/sda2 none swap sw
/dev/sdb2 none swap swap sw
Disable Swap Partition on the Defective Device
Deactivate the swap partition on the defective hard drive so that it can be replaced. To do this, enter the following command:
[root@host~]# swapoff PATH_OFTHE_HARD_DRIVE
Example:
[root@host~]# swapoff /dev/sdb2
Please Note
If the swap partition on the defective hard drive is not deactivated and the hard drive is replaced, the swap partition in /proc/swaps receives the status Deleted.
Requesting the Hard Drive Replacement
The defective hard drive can now be replaced. To request the replacement, please contact IONOS Customer Support.