Online Capacity Expansion (Gone South) with Openfiler and a 3Ware RAID card
If you've been reading my blog then you'd know that I use OpenFiler for my home storage needs. When I originally provisioned the system it was with a pretty beefy box. One thing I went short on though was storage. I purchased a high-end RAID card (3Ware 9590SE-16ML) but then proceeded to only install 3 drives (Western Digital RE2 500 GB). I did this for a bunch of reasons, but the most important are that, 1. hard drives cost money and they wern't currently needed, 2. hard drive prices go down, and 3. the raid controller I purchased supports Online Capacity Expansion (OCE).
A few days ago I noticed that my storage array was about 80% full. As such, I decided to order another 500GB RE2 (for $40 less than the original three, proving point #2) and test out 3Ware's OCE.
The steps below are what I had planned to do to get the system up and working with the new, larger volume.
- Install the extra drive (turn computer off if you don't have a hot-swap drive cage).
- Open 3DM2 in your web browser and log in as 'admin'.
- Click "Management > Maintenance".
- Check the check box next to your existing array and click "Migrate".
- In the pseudo-popup, select the new drive that you just added, verify the RAID level & stripe size, and then click "OK".
At this point I waited almost 37 hours for the migration to complete. I was expecting to be able to simply use the lvm tools ('pvresize', etc.) to enlarge the physical volume, volume group, and logical volume. Alternatively, I assumed that I could just create a second physical volume, etc. and add it to the existing volume group to accomplish basically the same thing (although this would have limited me in the long run since you can only have 4 primary partitions).
This is where I ran into trouble.
When I installed 2.2 a few months back I went with the 64-bit version and selected a GPT partition table so that I could eventually use a larger-than-2TB volume by taking advantage of OCE and only adding a drive when needed. As it turns out, this makes increasing the physical volume size all but impossible because fdisk can't see it to start (GPT uses a forged MSDOS partition table) and parted can't resize it becuase of the use of LVM (whch means that the physical volume that parted can see doesn't have a file system).
After 2 days of tinkering with the partition table and various versions of LVM & parted, I managed to completely break my GPT partition table. I've managed to "hotwire" the partiton table by rewriting it to think that the logical volume was actually the physical volume. Unfortunately, this means that the OpenFiler GUI is now all-but-useless. As such, I'll be reinstalling sometime over the next couple days. I've ordered another pair of 500GB drives that I'm going to add to the array so hopefully this will all be worked out before I need to expand again.
Oh, and here's a link to a thread on the OpenFiler forums with more information about this issue and what I did to break my system.