mjhjmtu

I have a main storage / iSCSI server where I want to keep all my data.

On the other side I have workstations and VMs where I want no data at all. However due to performance requirements I want to keep some local storage at workstations and VMs to serve as /boot and cache for the primary iSCSI volume.

I have not yet decided whether to use bcache or LVMs dm-cache so if there's any difference between them regarding this question, please mention them in your answer.

The question:

How can I install linux (ubuntu) such that my local storage would only serve as GRUB, /boot & iSCSI initiator mounting one specific volume on the target (no PXE involved). Cache too, of course. Ideally this would be installed to each workstation using command line from a live CD or something. For VMs I'd just copy the disk itself and then change the iSCSI volume to mount.

Well, this turned out to be something of a learning experience :)

Step 1

To make this work, I took Ubuntu network installer server image. It's just 55MB.

Step 2

The installation is text based, but no less functional as the desktop counterpart. It does have a minor difference of allowing you to specify iSCSI connection parameters directly within the installation, but you do have to choose manual partitioning. I'm not sure it's that helpful though as entering initiator info into the installer resulted in same LUN being mounted twice which is a bit unfortunate as that messes up (at least) LVM I'm installing next. So I just entered initiator ID from within the installer and then switched to console #2 (ctrl + alt + F2) and connected to the target manually from there. Returning back to the installer and manual disk partitioning I now had /dev/sda (my local storage) & /dev/sdb (the iSCSI volume).

Please note that a desktop installer (for ubuntu gnome edition) did not preinstall open-iscsi into the deploy and I ultimately gave up on it and just went with the server installer.

Step 3

Proceeding with manual partitioning, the next step is to establish base conditions for the SSD cache. In this case I decided to use LVM's dm-cache implementation, so for now I simply created a volume group in the iSCSI LUN and created a logical volume within it. This will be the root. Please note that having an existing logical volume on the iSCSI LUN does not display it in the installer's partition manager, so you might need to delete it first and create a new one before you're able to proceed.

Step 4

Finishing up the manual disk partitioning, I now created the partitions:

• sda1: /boot


• sda2: swap


• sdb:vg/lv: / (root)

And installed the OS into them. You get a choice of your favourite desktop flavour during installation, so there's no mission out on anything. You can even choose to only install necesities first and later restart tasksel as suggested in the link just above.

Step 5

This left me with a non-functional deployment for two reasons:

1. The iSCSI initiator ID burned into deployment's initramfs was not the one used to connect to the target.


2. The password information for CHAP authentication was also left out in the initramfs image

As a result, my newly installed Ubuntu dropped to initramfs prompt as it couldn't mount the iSCSI image containing root FS.

I'm not sure how I could fix this during the install phase, so I just fixed it with the following steps:

1. I verified what client ID my new deploy was using and created an appropriate ACL in my server's targetcli. At the same time I disabled CHAP authentication. This would have been better accomplished by simply invoking iscsistart from within initramfs prompt with the correct parameters, but at the time I simply didn't know that. Choose your favourite poison here.


2. After my client booted, I fixed the /etc/iscsi/initiatorname.iscsi with the correct initiator ID and /etc/iscsi/iscsi.initramfs with full target & authentication details. The parameter names for the latter file are ISCSI_INITIATOR, ISCSI_TARGET_NAME, ISCSI_TARGET_IP, ISCSI_TARGET_PORT, ISCSI_TARGET_GROUP, ISCSI_USERNAME, ISCSI_PASSWORD, ISCSI_IN_USERNAME, ISCSI_IN_PASSWORD. I got them here. Following the updates, I issued update-initramfs -u to update the boot configuration accordingly.

After this step I have a correctly setup system which boots and operates as I wanted from the start. There remain two steps that I haven't completed yet.

Step 6

The deployment has a "bug" where during shutdown, the network stack and iSCSI volume get shut down before the root filesystem gets unmounted. This results in shutdown hanging mid-way. When I find a fix for this, I'll update this step accordingly and proceed to the final step.

Step 7

After fixing the shutdown issue, I want to setup SSD caching as explained here & here.

I'm currently looking into LVM caching for two reasons:

1. I have set up bcache on my main server and while it's very powerful and configurable and reliable, I'm experiencing certain minor (but nonetheless annoying) issues with it.


2. LVM's caching leaves logical volumes' paths intact: you can enable / disable / remove cache at will with no changes in how you use the underlying volumes. bcache on the other hand creates a new mapping which needs to remain active even if you disable the cache itself. Well, technically, since I had to setup LVM just to be able to enable cache later on, I guess it's not really fair of me to say that bcache has another layer and LVM doesn't, is it?