Making a newish storage-server

Posted on
zpool zfs storage self-built tech

I have had my storage server for at least a couple of years now, in its current setup.

I built it to be a storage sever, so inside I has a ASUS TUF mb, AMD Ryzen 2200G, 16 GB RAM, 10 WD RED, 1 System SSD (Running Ubuntu server 20.10) and Fractal design PSU 650W. All living inside a Fractal Design Define 5 case.
The disks are setup as a zfs mirror using ZoL 0.86, the version built into Ubuntu.

The new one is getting an update, new motherboard ASUS ROG STRIX B450-F GAMING (Mostly for the three PCI-E ports), more and faster memory (64 GB 3200 MHz), Fractal Design ION+ PSU 750W all will fit inside a Fractal Design Meshify 2.

I do hope that I can reuse the sata-psu cables that is why I went with the same brand on the PSU.

It is possible to refit the case to hold up to 14 2,53,5” disks. That is what I going for.

Update 1: 20210201
Added an backup disk to the server, that is the disk with the red sata-cable… the rest is the server in its full glory.

So at the moment I am taking backup of parts of the filesystem that is not shipped to an offisite backup.

Prepping the new OS…

10.7 will be installed with ZFS on Linux (ZoL) 2.0.
Backup not ready yet….
Backup finished it took almost 3 hours to backup everything and lucky me the backup disk got filled up until 1% left on it…
So now I have really filled my Trello board with what I now think is everything that I need to do.

Using Trello to clear the trees so I can see the forest is a good way to work.
By putting down everything that comes to mind let you think even deeper och in a different direction, at least for me.

This led to that I realized that I need to identify the disks in my zpool.

For that I used: blkid, ls -al /dev/disk/by-id/ and zpool status to create a spreadsheet so I now know which disk is where when I start to disassemble the old server.
I have also discovered that I might have ordered a bit many disk holders for the new server. There are 6 in the case box, and I need 10 in total, that is 4.
So I ordered 4 packages, forgot that they come 2 in every package…
Oh well - I do order from one of the best computer businesses in Sweden, https://inet.se - I can whole heartily recommend them.

Well, I was happy that everything went so smoothly yesterday that without any regrets started to disassemble the old server. When I pulled the last cord it hit me like a ton of bricks… I didnt do the zfs export command.

!!!! NO !!!!
Oh well, so today I will reassemble the old server, start it, do a zfs export and disassemble it again.
So the suspense is cruel - what did happen?!
Without going into too many details about why the old servers PSU gave up but it all happened before I had a chance to run the zfs export command but according to what I have read and got help with from the zfs-community at Reddit - this should not be an issue.

So yesterday nigth meant to prepare everything for today. That meant assembling the new motherboard as far as possible, prep the case and the disks.

64 GB of RAM at 3200 Mhz is one of the biggest changes to the system.
Ad those three PCI-E lanes will come in handy later.

The case is prepped as far as possible, I do really like the ability to remove pretty much everything to get to the bare metal. Well done Fractal Design!

And lastly but not unimportant in any way.

Moving the 10 disks from their old to their new tray. Almost like knitting to a TV-show.
So today I will try to film my build - see you later!

Woha that was quite a bit later… the day took an unprepared turn.
But first a build image… this was the first build I made, yes first.

When I tried to start this first build all I heard was a ‘click’ and then nothing. That click was a suspicious sound. Searched for it and it seemed like it meant that the PSU stops to protect itself.

And why does a PSU do that? Something is shorting or something like that bad. Tried to see if I could see the error without disassembling everything…
No, I could not so I had to pull everything out again. And than the error searching began. PSU, MB, Proc, and one memory stick, it starts! Yes! Those parts are operational at least!

Added the fans, it is working.
Added the SSDs, it is working.

Then I realized that the back fan wasn’t spinning and that was hooked up to the fan control unit that comes with the case. So I added SATA power to it and hit start… then I saw a small stream of smoke! GAH! Shut it down as fast as possible… the sata-cord for the fan control was WARM, way to warm. What the funk was my thought is the fan controller the culprit in my build?

So I added as much stuff I could, bypassed the fan controller, added all the fans, and pushed start. No issue what so ever!

With a high sigh of relief, I was happy to locate the error, called the store where I bought it told the story, they wanted to send me a new case, but that was way too much work to change the case. So I asked if I was able to replace the fan controller only and that was ok they said. So now Fractal Desing is gonna send me a new fan controller.

But still, the second build went so much better, it usually does.

So at last it was time to install the system. But first dinner with the kids.

Well… where to begin?
Oh yeah - when I was to install the OS, Debian, it failed with Cannot install Grub. The boot loader. I searched the net and didn’t find any good solutions, tried a few but all failed.
Got rid of my pride and tried and failed with: LinuxMint Cinnamon and Debian Edtion. They did install and booted to that was a fall forward but there were other issues that said a big NO to those being a server OS but as a client, they rule at that.

So it ended with me install Ubuntu server again, oh the fun with configuring netplan.

Then I installed ZoL 2.0.2, and tried ‘sudo zpool import tank’

FAIL! Panic! What the fffff…
‘sudo zpool import -f tank’

Didn’t work…

‘sudo zpool import -a’

‘cannot import ‘tank’: pool was previously in use from another system.
Last accessed by storage (hostid=e7204e30) at Tue Feb 2 17:54:51 2021
The pool can be imported, use ‘zpool import -f’ to import the pool.’
No no no!
‘sudo zpool import -D -f tank’

no pools available to import

Then I ran this just in pure desperation:

*sudo zpool import -f’
And got this:

pool: tank

 id: 11331981177896442112

state: UNAVAIL

status: The pool was last accessed by another system.

action: The pool cannot be imported due to damaged devices or data.

see: http://zfsonlinux.org/msg/ZFS-8000-EY

config:
tank UNAVAIL insufficient replicas
mirror-0 ONLINE
ata-WDC_WD20EFRX-68AX9N0_WD-WMC301231995 ONLINE
ata-WDC_WD20EFRX-68AX9N0_WD-WMC301319536 _LINE
mirror-1 UNAVAIL insufficient replicas
ata-WDC_WD2003FYYS-18W0B0_WD-WMAY05223379 UNAVAIL
ata-WDC_WD2003FYYS-18W0B0_WD-WCAY00257374 UNAVAIL
mirror-2 ONLINE
ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M2TAS4TK ONLINE
ata-WDC_WD20EFAX-68FB5N0WD-WXA1AC8J2YR7 ONLINE mirror-3 ONLINE
ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M4AUPTJE ONLINE
ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M0UX4964 ONLINE mirror-4 ONLINE
scsi-SATA_WDC_WD40EFRX-68N_WD-WCC7K0XN9YCL ONLINE
scsi-SATA_WDC_WD40EFRX-68W_WD-WCC4E5LLH82H ONLINE

Two disk are not present! Ehhhh ok… searched for the disks using fdisk and grep and they are not present in the machine. Explains a lot… so that means I have to open the server and check cables…

It seems that the two WD Black disks are not with me anymore… but that 2 disks should break at the same time is kind of unheard of. So come Monday I will dismount them and investigate them outside the server.
I have learnt my lesson years ago that storage is not backup - so I have backup of everything. It is very nice not feeling the panic to retrieve anything something.

So some headway has been done today, after some issues with creating the zfs pool it is finally up.
So now the restoring of backup is to begin, first I started the offsite backup.

So that concludes this post.

Thanks for following me - hope I could learn you something.