Ttyjfyk

I had a nice Ubuntu 18.04 system up and running with many different internal SATA drives and rock solid for several months, and all were happy in the land. Then one day, there was a simple power outage. No problem, we've been through them before. Once the power came back on, the server didn't respond to SSH; sometimes I have to go down to the basement and hit the button on the box to turn it back on. I did that, and again after a few minutes, I still can't access via SSH.

So now I'm down in the basement with a little monitor and keyboard hooked up to the box and I can see that the Ubuntu logo is up there quite a while after reboot so I hit the ⬆ key and watch the boot loader thingy and it seems where its timing out is with something like:

A start job is running for dev-disk-byx2duuid-914d3b77x2d06c4x2d4514x2d8feex2d1fc6eb81bbd9.device (51s / 1min 30s)

There are actually two of these errors at first, both seeming to time out after reaching 1min 30s. The other start job that times out appears to reference UUID=a158e6ec-1433-454c-9cd2-10f7306fde82. So I search around and read about how the UUIDs in the job that is timing out correspond to some of the UUIDs I specify in /etc/fstab to automatically mount my internal hard drives upon boot.

After 1:30, I hit enter to get to a root command prompt and run vim /etc/fstab and comment out the lines that correspond to my 2 errors so now the file looks like this:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda2 during installation
UUID=6f90b496-401e-475f-add0-3c6d3bcae7a0 / ext4 errors=remount-ro 0 1
# /boot/efi was on /dev/sda1 during installation
UUID=BFE0-55D4 /boot/efi vfat umask=0077 0 1
/swapfile none swap sw 0 0
UUID=bc731122-7e4e-47d9-b6a5-db1f703f96a8 /media/Tre ext4 defaults 0 0
UUID=6c7b175d-b80b-4069-bbbe-f82aeb302200 /media/Sam ext4 defaults 0 0
#UUID=a158e6ec-1433-454c-9cd2-10f7306fde82 /media/Hex ext4 defaults 0 0
#UUID=914d3b77-06c4-4514-8fee-1fc6eb81bbd9 /media/Wes ext4 defaults 0 0

After saving the file, I run reboot and Ubuntu reboots pretty quickly. It doesn't hand and I get to the login screen. I crawl out of the deep dark basement server closet and SSH back in from the couch.

I use blkid and discover the UUID on the drive I call Wes appeared to be different than what I had in /etc/fstab, so I took a backup and edited that UUID to the one from blkid and uncommented that line. Another reboot and now I've got Wes back. So now I am just missing the big 6TB drive I call Hex and my /etc/fstab looks like:

# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda2 during installation
UUID=6f90b496-401e-475f-add0-3c6d3bcae7a0 / ext4 errors=remount-ro 0 1
# /boot/efi was on /dev/sda1 during installation
UUID=BFE0-55D4 /boot/efi vfat umask=0077 0 1
/swapfile none swap sw 0 0
UUID=bc731122-7e4e-47d9-b6a5-db1f703f96a8 /media/Tre ext4 defaults 0 0
UUID=6c7b175d-b80b-4069-bbbe-f82aeb302200 /media/Sam ext4 defaults 0 0
#UUID=a158e6ec-1433-454c-9cd2-10f7306fde82 /media/Hex ext4 defaults 0 0
UUID=bc731122-7e4e-47d9-b6a5-db1f703f96a8 /media/Wes ext4 defaults 0 0

If I uncomment the line for Hex, I get the endless loop of waiting 1:30 for that same job to time out. If I use journalctl -xe to view the logs and go to what I think is where things go wrong, I see a red error like:

zen systemd[1]: Timed out waiting for device dev-disk-byx2duuid-a158e6ecx2d1433x2d454cx2d9cd2x2d10f7306fde82.device

which would seem to correspond to the same Hex drive.

Fearing the drive was dead, I pulled the box from the closet and opened it up and removed that particular hard disk drive. I took it to my desk and put it in a SATA to USB interface and supplied power. The drive began spinning and I hooked it up to my Mac laptop. When I opened Disk Utility, I could see the HDD, but it said it was only 1.8TB and I could not see the partitions.

I figured that part might be right as I seem to remember having to do something special to format that 6TB drive and maybe the Mac just can't see it? Anyways, I'm encouraged by seeing the drive at all and decide to go put it back in the server.

I read more and search around more and I'm in this loop. I can comment out the /etc/fstab entry and get into the system via SSH, or I can uncomment it and the reboot hangs. If I cd into /media/Hex I can see a few of the top level folders and files, but the vast majority of the file structure appears gone, or invisible to me.

How to get Hex back to normal?

Best case: the system is just trying to run a journal recovery and/or filesystem check on the Hex filesystem and it's taking more than 1 min 30s because the filesystem is so big. In this case it might be better to run the filesystem check manually once the rest of the system is up, see below for instructions.

Worst case: the disk might be failing at the hardware level.

If you can see some files and folders in /media/Hex while its line is commented out in /etc/fstab and you haven't mounted it manually, then some program may have written into the empty directory that is supposed to be just a mount point for Hex, while the actual Hex filesystem was unmounted. That has nothing to do with the state of the files in the actual Hex filesystem, but can be a bit of a pain to merge back into the real Hex once the current problem with the filesystem is fixed.

Leave the /etc/fstab line for Hex commented out for now. With the system up and Hex connected to it, use the blkid command as you've done before to identify the device name Hex currently has. Once you know that, you can start further diagnostics.

Assuming that the device that contains the Hex filesystem is named something like /dev/sdX1 (where X is a letter unknown to me - you must figure it out yourself), then drop out the partition number to know the corresponding whole-disk device, i.e. just /dev/sdX. It is also possible to initialize an entire disk for a filesystem without using any partition table at all - in that case, blkid will report the filesystem as existing directly on the whole-disk device /dev/sdX.

A good first step would be to run sudo smartctl -H -a -f brief -l error /dev/sdX: it should report the overall health status of the disk, the SMART attributes and any hardware-level errors the disk itself might have logged. If these indicate the disk is OK at the hardware level, you can proceed. (If you are unsure as how to interpret the result, please edit your original question and add the output to it.)

If the disk indicates a hardware failure, you'll need to make a decision: if a failing disk contains important data, it might be best to stop trying to fix it yourself and send the disk to a data recovery professional. A professional recovery might cost something, but has the highest chance of getting the data successfully recovered. If you choose to do this, don't do anything further to the disk yourself: if the disk is physically failing, keeping the disk powered up might make the problem worse and make the recovery job harder.

If the disk is OK at the hardware level, or the data in it is "not worth the money of professional recovery but it would be nice to have it back", the next step would be to understand the partitioning situation. Since you said it's a 6 TB disk, it probably uses GPT partitioning. Any of these commands should display the partition layout:

sudo parted /dev/sdX print
sudo gdisk -l /dev/sdX
sudo fdisk -l /dev/sdX

(I think the fdisk of Ubuntu 18.04 is new enough to understand GPT partitioning, but if it reports something like a 1.8 TB partition while the other commands show the full 6 TB, that might not be a real problem.)

Please edit your original question to add this partitioning information. The next steps will depend on the results of these checks.

If the disk has a SMART failure indication, the next step would be buying a new disk of equal or greater size, and trying to copy the entire contents to the new disk as soon as possible, perhaps using ddrescue or a similar tool. Once that is done, your data should be safe from further deterioration.

If the disk's SMART diagnostics are OK and the filesystem can be recognized, it might still be a good idea to first clone the disk to a new one before proceeding, just in case. Then, you might try running a filesystem check on the device that is reported to contain the filesystem: something like sudo fsck.ext4 -f -C0 /dev/sdX1. On a large disk, this may take quite a while: the -C0 option tells the filesystem checker to provide a completion percentage while it's running.

If the filesystem check is successful, the next step is to try and mount it manually: for example, sudo mount /dev/sdX1 /media/Hex. If it says /media/Hex is in use, you must stop any services that might be using the /media/Hex mountpoint while you do the mount.

If /media/Hex looks normal after mounting it, you can have your sigh of relief, unmount it manually (sudo umount /media/Hex), uncomment its line in /etc/fstab and reboot your system: now it should again boot normally as the filesystem has been checked and cleanly unmounted.

To clean up any files left in the mountpoint directory (that is now hidden "under" the real Hex filesystem) after the real Hex filesystem is mounted, you can do this:

sudo mkdir /mnt/Hex_oops
sudo mount --bind / /mnt/Hex_oops
cd /mnt/Hex_oops/media/Hex

... and then see if the files and directories in /mnt/Hex_oops/media/Hex are of any importance. If they are, you could now move them to their correct place in the real Hex filesystem; if they are just empty directories created automatically by some application, you can delete them (as they are uselessly using up space in your root filesystem). Then remove this temporary arrangement:

cd /
sudo umount /mnt/Hex_oops
sudo rmdir /mnt/Hex_oops

Replace defaults option with noauto,x-systemd.automount in your /etc/fstab:

UUID=a158e6ec-1433-454c-9cd2-10f7306fde82 /media/Hex ext4 noauto,x-systemd.automount 0 0

From Archlinux wiki: Automount with systemd

In case of a large partition, it may be more efficient to allow services that do not depend on it to start while it is checked by fsck. This can be achieved by adding the following options to the /etc/fstab entry of the partition:

noauto,x-systemd.automount

This will fsck and mount the partition only when it is first accessed, and the kernel will buffer all file access to it until it is ready. This method can be relevant if one has, for example, a significantly large /home partition.