Building a 18TB Ubuntu RAID6 fileserver for a Mac network

I was recently tasked with setting up a new fileserver for a network of Macs. We had previously used a Mac Pro and an external firewire drive array for this, but it wasn't very fast or reliable, and was only RAID5 - we wanted to move to having two redundant drives rather than just one. Netatalk seems to have matured as an AFP (Apple Filing Protocol) server, and Linux software RAID is very fast and reliable on modern hardware, so this seemed like a cost-effective option.

Hardware
XCase RM420 20-bay 4U rackmount case
CiT 850W ATX PSU
ASRock P67 Extreme6 mobo (6 6gbit + 4 3gbit SATA ports, dual LAN)
Intel i5-2500k Sandy Bridge proc
Cheap NVidia card
8gb RAM
8x 3TB Seagate Barracuda hard drives
2x SFF-8087 to SATA reverse breakout cables

Assembly
Everything was easy to put together, and almost worked fautlessly - it turned out that the 2nd bay from the left on the bottom SATA backplane didn't work reliably - Linux dropped the SATA link speed down to 1.5gbit and then encountered sporadic read errors. I have yet to take the backplane out and try and get a replacement.

All ten SATA ports were set to AHCI mode to allow hot-swapping and other such cleverness.

Installation

The 3TB disks use 4k sectors (as opposed to the traditional 512 byte sectors), which caused confusion at various stages of the installation process. I used cfdisk to partition the drives, but that did not align the start of the partitions with the 4k sectors which upset the Ubuntu installer. Eventually using fdisk to create the partitions seemed to work.

After USB-booting the Ubuntu installer, all 8 disks were tested (hdparm -tT) and individually partitioned with a 20G type 0xfd (Linux RAID autodetect).

I then planned to create a RAID6 (giving (8-6)*20 = 120gb) for the system, and use the rest of the disks for the share partition(s) later. This would speed up boot and make offline disk checking possible.

Attempt #1: BIOS-booting and msdos partition table

The RAID was created with:

mdadm --create /dev/md0 --chunk=16 --level=6 --raid-devices=8 /dev/sd{a,b,c,d,e,f,g,h}1

I opted for 16k chunk size for the system, assuming this would speed access to lots of small files (a larger chunk size would probably make sense for the actual storage partitions which will have lots of huge media files on). I then partitioned the /dev/md0 device, into a 110gb Linux partition (leaving room for 10gb swap should I decide to use it).

The Ubuntu installer was happy to install on /dev/md0p1 (using it as mount point /), but failed to install GRUB to it, so the system wouldn't boot. The resulting installation was also lacking mdadm, so was unable to bring up the RAID even if it had booted (I was using the Desktop edition of Ubuntu, which might have been why).

Fixing this was somewhat tricky. Booting onto the USB stick, it went something like:

mdadm --assemble /dev/md0 /dev/sd{a,b,c,d,e,f,g,h}1
mount /dev/md0 /target
mount --bind /dev /target/dev
mount --bind /proc /target/proc
mount --bind /sys /target/sys
sudo chroot /target
echo "nameserver 8.8.8.8" > /etc/resolv.conf
apt-get install mdadm
grub-install /dev/sda
grub-install /dev/sdb
etc...

Attempt #2: EFI boot and GPT partition table

The msdos partition table format is old fashioned and strange - you can only have four primary partitions, and must use the strange second-level logical partitions for further ones. This machine was probably going to have quite a few partitions (primarily to allow RAID rebuilds to be done in order of importance), so the newer GPT format made sense. I turn, with GPT comes EFI boot instead of the older BIOS-boot method (you can in theory still BIOS-boot GPT partitions, but this requires a slightly strange fake-partition at the start of the disk to accomodate the bootloader).

I finally discovered parted the partitioning tool, which is much better than fdisk and cfdisk. Creating the partition table and partitions went something like (repeated 8 times):

$ sudo parted /dev/sda
unit mib
mklabel gpt
mkpart
... fat32, start 0, end 200
flags 1 boot on
mkpart
... ext4, start 200, end 8192
flags 2 raid on

Note the 200mb partition at the start of each disk with the boot flag set - this is the EFI partition. These were then mkfs.vfat'd, ready for the directory structure and bootloader image file that EFI requires.

I then installed Ubuntu off a USB key booted in EFI mode (chosen using the BIOS boot menu). For some reason, Ubuntu will only install to be EFI bootable if it is booted in EFI mode. The RAID6 was set up in the same way as earlier. The installer detected the first EFI partition and copied the x64 .efi file into the /EFI/ubuntu folder in the EFI disk, and the system installed and rebooted from the RAID6 successfully.

The installer had only set up the EFI boot partition on the first hard disk. I wanted it to be possible to boot this machine with any two drives missing, so I needed to duplicate the folder structure and copy the .efi file to all of the other EFI boot partitions, which was easy enough. However, the next step was less obvious - you need to use the efibootmgr program to add each of the other drives to the EFI boot menu (stored in Flash somewhere). The machine will now boot with any disk missing - success!

Jan 5th 2013 | Permalink

Instruction manual / Recipe book for Prima ABM6 Breadmaker

It seems Prima are no more, so the all-important recipe book for the ABM6 is now harder to find.

Download here!

Jun 4th 2012 | Permalink

Writing HTML emails for maximum compatibility

Getting HTML emails to look right in many email clients simultaneously is a nightmare, here are some tips:

1. Make the images as large as possible (when cutting them out), to maximise the likelihood of the ALT text fitting.

2. Style A tags twice (with an inner span), like so: <a href="..." style="color: #123456;"><span style="color: #123456">link</span></a>

3. Style ALT text containers so that the ALT text will be styled in some clients.

4. Use #FFFFFE instead of #FFFFFF to stop Gmail removing it.

5. Use lots of nested tables (ugh).

Jun 20th 2011 | Permalink

Installing Mac OSX (Snow Leopard) from an external hard drive

The surprisingly-flexible Mac OSX will boot and install from an external drive (on a MacBook Pro at least), possibly useful if you've got no DVD drive (or a 6 gig .dmg and no dual-layer DVD-Rs!).

Install dmg2img, and perform the following:

dmg2img -i installer_cd_image.dmg -o /dev/sde

where /dev/sde is the USB drive. This will overwrite the MBR and partition table. Then plug the USB drive into the Mac, hold Alt whilst booting, select the external drive, et voila.

Feb 14th 2010 | Permalink