Linux and iSCSI
References: http://www.open-iscsi.org/docs/README , manpages
This section is a quick howto on getting Linux, open-iscsi and iscsi-target play together to configure an iSCSI storage system. Please let me know if you have any comments. The cheap way is to run Linux iSCSI target (the "disk" or "storage device" - similar to NFS server) on linux machine with enough free disks and run the open-iscsi initiator (the system that uses this disk - similar to NFS client) Ofcourse, if you wish to use hardware based iSCSI devices for superior performance, we can acquire, install and support IBM iSCSI devices. The following configuration assumes either Debian 4.x or Ubuntu Feisty Fawn or newer systems
* On the server, download and compile iSCSI Enterprise Target (
root@fs:/usr/local/src/iscsitarget-0.4.15# uname -a
Linux fs 2.6.20-16-generic #2 SMP Thu Jun 7 20:19:32 UTC 2007 i686 GNU/Linux
apt-get install build-essential linux-headers-generic libssl-dev
wget http://jaist.dl.sourceforge.net/sourceforge/iscsitarget/iscsitarget-0.4....
tar zxvf /root/iscsitarget-0.4.15.tar.gz
cd iscsitarget-0.4.15
root@fs:/usr/local/src/iscsitarget-0.4.15# make
make -C usr
make[1]: Entering directory `/usr/local/src/iscsitarget-0.4.15/usr'
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o chap.o chap.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o event.o event.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o param.o param.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o plain.o plain.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o isns.o isns.c
cc ietd.o iscsid.o conn.o session.o target.o message.o ctldev.o log.o chap.o event.o param.o plain.o isns.o -o ietd -lcrypto
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include -c -o ietadm.o ietadm.c
cc ietadm.o param.o -o ietadm
make[1]: Leaving directory `/usr/local/src/iscsitarget-0.4.15/usr'
make -C /lib/modules/2.6.20-16-generic/build SUBDIRS=/usr/local/src/iscsitarget-0.4.15/kernel modules
make[1]: Entering directory `/usr/src/linux-headers-2.6.20-16-generic'
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/tio.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/iscsi.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/nthread.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/wthread.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/config.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/digest.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/conn.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/session.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/target.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/volume.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/iotype.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/file-io.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/null-io.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/target_disk.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/event.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/param.o
CC [M] /usr/local/src/iscsitarget-0.4.15/kernel/block-io.o
LD [M] /usr/local/src/iscsitarget-0.4.15/kernel/iscsi_trgt.o
Building modules, stage 2.
MODPOST 1 modules
CC /usr/local/src/iscsitarget-0.4.15/kernel/iscsi_trgt.mod.o
LD [M] /usr/local/src/iscsitarget-0.4.15/kernel/iscsi_trgt.ko
make[1]: Leaving directory `/usr/src/linux-headers-2.6.20-16-generic'
root@fs:/usr/local/src/iscsitarget-0.4.15# make install
install: creating directory `/lib/modules/2.6.20-16-generic/kernel/iscsi'
`kernel/iscsi_trgt.ko' -> `/lib/modules/2.6.20-16-generic/kernel/iscsi/iscsi_trgt.ko'
depmod -aq
`usr/ietd' -> `/usr/sbin/ietd'
`usr/ietadm' -> `/usr/sbin/ietadm'
`etc/initd/initd.debian' -> `/etc/init.d/iscsi-target'
`etc/ietd.conf' -> `/etc/ietd.conf'
`etc/initiators.allow' -> `/etc/initiators.allow'
`etc/initiators.deny' -> `/etc/initiators.deny'
`doc/manpages/ietadm.8' -> `/usr/share/man/man8/ietadm.8'
`doc/manpages/ietd.8' -> `/usr/share/man/man8/ietd.8'
`doc/manpages/ietd.conf.5' -> `/usr/share/man/man5/ietd.conf.5'
install: creating directory `/usr/share/doc/iscsitarget'
`ChangeLog' -> `/usr/share/doc/iscsitarget/ChangeLog'
`COPYING' -> `/usr/share/doc/iscsitarget/COPYING'
`README' -> `/usr/share/doc/iscsitarget/README'
`README.vmware' -> `/usr/share/doc/iscsitarget/README.vmware'
root@fs:/usr/local/src/iscsitarget-0.4.15#
make install
Now configure :
root@fs:/usr/local/src/iscsitarget-0.4.15# grep -v '#' /etc/ietd.conf
Target iqn.2007-06.com.vsa-services:fs.lvdevvm
Lun 0 Path=/dev/vgfsmd1/lvdevvm,Type=fileio
Alias lvdevvm
/etc/init.d/iscsi-target start
We continue configuring the "client" First conduct a scan:
root@dev:/var/vm# iscsiadm --mode discovery --type sendtargets --portal 10.1.2.8
10.1.2.8:3260,1 iqn.2007-06.com.vsa-services:fs.lvdevvm
root@dev:/var/vm#
Add this target to the open-iscsi DBM database so that it is available upon subsequent restarts/reboots:
root@dev:/var/vm# iscsiadm -m node -o new -n lvfsdevvm -p 10.1.2.8 -T iqn.2007-06.com.vsa-services:fs.lvdevvm
new iSCSI node record added
root@dev:/var/vm# ls
At this stage, we have created a local definition of the iSCSI device which looks like:
root@dev:/var/vm# cat /etc/iscsi/nodes/iqn.2007-06.com.vsa-services:fs.lvdevvm/10.1.2.8\,3260
node.name = iqn.2007-06.com.vsa-services:fs.lvdevvm
node.transport_name = tcp
node.tpgt = -1
node.active_conn = 1
node.startup = manual
node.session.initial_cmdsn = 0
node.session.auth.authmethod = None
node.session.timeo.replacement_timeout = 120
node.session.err_timeo.abort_timeout = 10
node.session.err_timeo.reset_timeout = 30
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 0
node.session.iscsi.MaxConnections = 1
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
node.conn[0].address = 10.1.2.8
node.conn[0].port = 3260
node.conn[0].startup = manual
node.conn[0].tcp.window_size = 524288
node.conn[0].tcp.type_of_service = 0
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.active_timeout = 5
node.conn[0].timeo.idle_timeout = 60
node.conn[0].timeo.ping_timeout = 5
node.conn[0].timeo.noop_out_interval = 0
node.conn[0].timeo.noop_out_timeout = 0
node.conn[0].iscsi.MaxRecvDataSegmentLength = 131072
node.conn[0].iscsi.HeaderDigest = None,CRC32C
node.conn[0].iscsi.DataDigest = None
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No
root@dev:/var/vm#
Now login to the iSCSI portal as follows, after which we can "see" the SCSI device:
root@dev:/var/vm# iscsiadm -m node -l -p 10.1.2.8 -T iqn.2007-06.com..vsa-services:fs.lvdevvm
root@dev:/var/vm# fdisk -l
Disk /dev/sda: 36.7 GB, 36703934464 bytes
255 heads, 63 sectors/track, 4462 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 1216 9767488+ 83 Linux
/dev/sda2 1217 1459 1951897+ 82 Linux swap / Solaris
/dev/sda3 1460 2432 7815622+ 83 Linux
/dev/sda4 2433 4462 16305975 83 Linux
Disk /dev/sdb: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdb doesn't contain a valid partition table
root@dev:/var/vm#
/dev/sdb is the 100GB iSCSI device. We can install GPFS or create partitions using fdisk and format with ext3 and use it! For example:
root@dev:/var/vm# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 13054.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): p
Disk /dev/sdb: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-13054, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-13054, default 13054):
Using default value 13054
Command (m for help): p
Disk /dev/sdb: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 13054 104856223+ 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
root@dev:/var/vm# partprobe
apt-get -y xfsprogs
root@dev:/var/vm# mkfs.xfs /dev/sdb1
meta-data=/dev/sdb1 isize=256 agcount=16, agsize=1638378 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=26214048, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal log bsize=4096 blocks=12799, version=1
= sectsz=512 sunit=0 blks
realtime =none extsz=4096 blocks=0, rtextents=0
root@dev:/var/vm# sync
root@dev:/var# mount /dev/sdb1 vm
root@dev:/var# df -hl|grep vm
/dev/sdb1 100G 544K 100G 1% /var/vm
root@dev:/var#
USB Storage (Thumbdrives etc)
Linux handles USB disks via the SCSI storage subsystem. In the following screenshots, "k4test" is the user who is NOT logged into the console but would like to use the thumbdrive. 1. Insert the USB thumbdrive. 2. Check linux system messages as follows:
anand@laptop-aries5672:~$ dmesg|tail
[15104.472000] sdb: Write Protect is off
[15104.472000] sdb: Mode Sense: 0b 00 00 08
[15104.472000] sdb: assuming drive cache: write through
[15104.476000] SCSI device sdb: 1967104 512-byte hdwr sectors (1007 MB)
[15104.476000] sdb: Write Protect is off
[15104.476000] sdb: Mode Sense: 0b 00 00 08
[15104.476000] sdb: assuming drive cache: write through
[15104.476000] sdb:
[15104.600000] sd 2:0:0:0: Attached scsi removable disk sdb
[15104.600000] sd 2:0:0:0: Attached scsi generic sg2 type 0
anand@laptop-aries5672:~$
So I know that my 1GB USB drive is sdb (or /dev/sdb) and has NO partitions on it
anand@laptop-aries5672:~$ fdisk -l /dev/sdb
Disk /dev/sdb: 1007 MB, 1007157248 bytes
31 heads, 62 sectors/track, 1023 cylinders
Units = cylinders of 1922 * 512 = 984064 bytes
Device Boot Start End Blocks Id System
3. Oops, when I plug it in, it is auto mounted!
anand@laptop-aries5672:~$ df -h|grep sdb
/dev/sdb 1.3G 4.1M 1.3G 1% /media/disk
anand@laptop-aries5672:~$
4. If I am not the console user, I cannot unmount and remount since I do not own the device: see below: Switch to another user ID:
anand@laptop-aries5672:~$ su - k4test
Password:
Try to unmount: ERROR!
k4test@laptop-aries5672:~$ pumount /media/disk
Error: device /dev/sdb was not mounted by you
k4test@laptop-aries5672:~$ logout
Unmount as ROOT: OK! (sudo elevates my normal account to rootly powers)
anand@laptop-aries5672:~$ sudo umount /media/disk
Password:
anand@laptop-aries5672:~$ df -h|grep sdb
anand@laptop-aries5672:~$
Now USB drive can be mounted by you as below:
k4test@laptop-aries5672:~$ pmount -s /dev/sdb usbdisk
k4test@laptop-aries5672:~$ ls -l /media/usbdisk/
total 4104
-rwx------ 1 k4test k4test 274432 2007-09-13 09:44 pscp.exe
-rwx------ 1 k4test k4test 282624 2007-09-13 09:44 psftp.exe
-rwx------ 1 k4test k4test 454656 2007-09-13 09:44 putty.exe
-rwx------ 1 k4test k4test 1733901 2007-09-13 09:44 winscp382setup.exe
-rwx------ 1 k4test k4test 1453056 2007-09-13 09:44 winscp402.exe
k4test@laptop-aries5672:~$
# You can unmount too:
k4test@laptop-aries5672:~$ pumount /media/usbdisk
k4test@laptop-aries5672:~$
o On Linux, not just devices, but even directories can be mounted on other directories, using the bind option:
mount --bind /var/home /home
Or from /etc/fstab:
/var/home /home none bind 0 0
o If LVM performance is bad, try setting a larger ReadAhead (from beowulf ml)
blockdev --setra 8192 /dev/${volumegroup}/${logical_volume}
Logical Volume management commands
Resizing an existing LVM volume is covered in a RedHat Knowledgebase article http://kbase.redhat.com/faq/FAQ_96_4842.shtm
# Create a physical volume for LVM:
pvcreate -v /dev/hdd
# create one or more volume groups:
vgcreate -v vg00 /dev/hdd
# start creating logical volumes:
lvcreate -L 5G -n lv00 vg00
# view PV info:
pvscan
# view volume group information:
root@c00:~# vgscan
Reading all physical volumes. This may take a while...
Found volume group "vg00" using metadata type lvm2
root@c00:~# lvscan
ACTIVE '/dev/vg00/lv00' [5.00 GB] inherit
The created volume can be formatted and used as usual.
mkfs.ext3 /dev/vg00/lv00
mount /dev/vg00/lv00 /data00
BLKID
If you are using Ubuntu Edgy and later, you will see that Ubuntu uses disk UUID and not /dev/hda1 etc to mount the filesystems.
You can use "blkid" command to view the unique ID assigned to HDD partitions by Linux, see example below:
haridwar:/usr/src/xen-3.0.2-2# blkid
/dev/sda1: UUID="397f2930-7b6f-4084-8a07-cee1d35dcb91" SEC_TYPE="ext2" TYPE="ext3"
/dev/sda2: UUID="b90ff46e-9505-4ce5-a23d-cb3f2e178189" SEC_TYPE="ext2" TYPE="ext3" /dev/sda5: TYPE="swap" /dev/md0: UUID="397f2930-7b6f-4084-8a07-cee1d35dcb91" SEC_TYPE="ext2" TYPE="ext3"
FDISK:
Note that with newer kernels, the kernel keeps an in-memory copy of partition table. When fdisk is used to edit the partition table, run: partprobe to update the kernel copy. Else kernel will complain!
Mounting remote filesystems with SSHFS
If you do not like NFS, you can install sshfs (apt-get install sshfs) and mount remote filesystems tunneled via SSH. SSHFS is a FUSE filesystem program. See example below:
sshfs 10.1.2.4:/home fileserver/home
anand@ariesduo:~/sshfs$ ls -l fileserver/home/
total 88
drwxrwxr-x 1 1001 users 4096 2006-09-12 12:16 admin
drwxr-xr-x 1 root root 4096 2006-11-06 20:10 backups
-rw-r--r-- 1 root root 132 2006-09-04 12:48 du.txt
drwxr-xr-x 1 root root 4096 2006-09-05 11:03 localsysbackups
drwxr-xr-x 1 root root 49152 2005-12-17 01:43 lost+found
anand@ariesduo:~/sshfs$