SCSI RDMA Protocol

The Linux SCSI Target Wiki

(Difference between revisions)
Jump to: navigation, search
m (Loading LIO Target)
m
 
(106 intermediate revisions not shown)
Line 1: Line 1:
{{Infobox software
{{Infobox software
-
| name                  = LIO Target
+
| name                  = {{Target}}
-
| logo                  = [[Image:Logo_Mellanox.png|180px|Logo]]
+
| logo                  = [[Image:Corp_Logo.png|180px|Logo]]
-
| screenshot            = Mellanox Technologies, Ltd.
+
| screenshot            = {{RTS screenshot|iSER}}
-
| caption                = Mellanox Infiniband SRP fabric module
+
| caption                = [[InfiniBand]] SRP fabric module
| collapsible            =  
| collapsible            =  
-
| author                = [http://www.linkedin.com/pub/vu-pham/6/93a/b40 Vu Pham]
+
| author                = {{Vu Pham}}<br/>{{Bart Van Assche}}<br/>{{Nicholas Bellinger}}
-
| developer              = Mellanox Technologies, Ltd.
+
| developer              = Mellanox Technologies, Ltd.<br/>{{RTS legal}}
| released              = {{RTS releases|SRP-Mellanox|initial_date}}
| released              = {{RTS releases|SRP-Mellanox|initial_date}}
| discontinued          = {{RTS releases|SRP-Mellanox|eol_date}}
| discontinued          = {{RTS releases|SRP-Mellanox|eol_date}}
Line 13: Line 13:
| latest preview version = {{RTS releases|SRP-Mellanox|preview_ver}}
| latest preview version = {{RTS releases|SRP-Mellanox|preview_ver}}
| latest preview date    = {{RTS releases|SRP-Mellanox|preview_date}}
| latest preview date    = {{RTS releases|SRP-Mellanox|preview_date}}
 +
| status                = {{RTS releases|SRP-Mellanox|status}}
| frequently updated    =
| frequently updated    =
| programming language  = C
| programming language  = C
Line 19: Line 20:
| size                  =
| size                  =
| language              =
| language              =
-
| status                = Beta
 
| genre                  = Fabric module
| genre                  = Fabric module
-
| license                = GNU General Public License
+
| license                = {{GPLv2}}
-
| website                = [http://www.mellanox.com/ www.mellanox.com]
+
| website                = {{RTS website}}
}}
}}
-
:''See [[Target]] for a complete overview over all fabric modules.''
+
:''See [[LIO]] for a complete overview over all fabric modules.''
-
An '''SCSI RDMA Protocol''' ('''SRP''', aka '''SCSI Remote Protocol''') fabric module (ib_srpt) for [http://www.mellanox.com Mellanox] HCAs is supported ([http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=tree;f=drivers/infiniband/ulp/srpt;hb=refs/heads/tcm_ib_srpt-38 ib_srpt.ko]). The [http://en.wikipedia.org/wiki/SCSI_RDMA_Protocol SCSI RDMA Protocol] defines a SCSI mapping onto the [http://en.wikipedia.org/wiki/InfiniBand Infiniband] architecture and/or functionally similar cluster protocols.
+
The '''SCSI RDMA Protocol''' ('''SRP''') is a network protocol that allows one computer system to access [[SCSI]] devices attached to another computer system via [[RDMA]].<ref name="SRP">[http://www.t10.org/drafts.htm#SCSI3_SRP ANSI T10 SRPr16a, www.t10.org].</ref><ref name="SRP-alternative">[http://web.archive.org/web/20080422013122/http://www.t10.org/ftp/t10/drafts/srp/srp-r16a.pdf ANSI T10 SRPr16a, web.archive.org]</ref>
-
The use of RDMA generally allows higher throughput and lower latency than TCP/IP based communication. RDMA is only possible with network adapters that support RDMA in hardware. Examples of such network adapters are InfiniBand HCAs and 10&nbsp;GbE network adapters with [[iWARP]] support. While the SRP protocol has been designed to use RDMA networks efficiently, it is also possible to implement the SRP protocol over networks that do not support RDMA.
+
== Overview ==
-
SRP is easier to implement than iSER, but iSER offers more management functionality, e.g. the target discovery infrastructure enabled by the iSCSI protocol.
+
SRP was designed to use RDMA networks efficiently. RDMA allows lower latencies and higher throughput than TCP/IP protocols, but requires network adapters with native RDMA support, e.g., [[HCA]]s for [[InfiniBand]], [[RNIC]]s on "lossless" ([[DCB]]) Ethernet, or [[TOE]] [[NIC]]s with [[iWARP]] for standard Ethernet.
-
== Status ==
+
SRP is based on the [[SCSI]] protocol, which is a point-to-point protocol with corresponding design limitations. In contrast, [[iSER]] is based on [[iSCSI]], and thus better accommodates modern network requirements, including complex topologies, multipathing, target discovery, etc. Hence, iSER is most likely the best choice for [[InfiniBand]] networks going forward.
-
The SRP driver is currently being tested and optimized.
+
SRP never became an official standard: the latest draft of the SRP protocol, revision 16a, dates from July 3, 2002.<ref name="SRP"/>
-
== Specifications ==
+
The InfiniBand/SRP fabric module ({{RTS releases|SRP-Mellanox|module_repo}}, {{RTS releases|SRP-Mellanox|module_info}}) for the {{Target}} was released with Linux kernel {{RTS releases|SRP-Mellanox|kernel_ver}} on {{RTS releases|SRP-Mellanox|initial_date}}.<ref>{{RTS releases|SRP-Mellanox|kernel_rel}}</ref>
-
SRP was not approved as an official standard. The following specifications are available as available as [http://www.t10.org/drafts.htm T10 Working Drafts]:
+
== targetcli ==
-
* '''SCSI RDMA Protocol''' ('''SRP'''): SRP defines a SCSI protocol mapping onto the InfiniBand (tm) Architecture and/or functionally similar cluster protocols. ANSI INCITS 365-2002. Status: Final Draft. 7/3/2002
+
''[[targetcli]]'' from {{RTS full}} is used to configure InfiniBand targets. ''targetcli'' aggregates {{T}} service modules via a core library, and exports them through an API, to provide a unified single-node SAN configuration shell, independently of the underlying fabric(s).
-
== Setup ==
+
{{Ambox| type=info| head=[[LIO]] Admin Manual| text=The {{LIO Admin Manual}} provides comprehensive background and many examples on using ''targetcli'' and on programming the [[Datera]] library.}}
-
This code won't be included in Linux before kernel v2.6.40. However using Linux >= v2.6.38, this and other HW target mode modules will be able to function. The installation and setup instructions are below. Please just contact us for [[RTSadmin]].
+
=== Cheat sheet ===
-
=== Getting the sources ===
+
{| class="table-left"
 +
|- bgcolor=#a0c0f0
 +
! Command
 +
! Comment
 +
|-
 +
| ''/backstores/iblock create my_disk /dev/sdb''
 +
| Create the LUN ''my_disk'' on the block device ''/dev/sdb''
 +
|-
 +
| ''/ib_srpt create <WWPN>''
 +
| Create an SRP target
 +
|-
 +
| In /ib_srpt/<WWPN>:<br/>''luns/ create /backstores/iblock/my_disk''
 +
| Export the LUN ''my_disk''
 +
|-
 +
| In /ib_srpt/<WWPN>:<br/>''acls/ create <Initiator WWPN>''
 +
| Allow access for the initiator at ''<WWPN>''
 +
|-
 +
| ''/saveconfig''
 +
| Commit the configuration
 +
|-
 +
|}
-
==== Initial build: git setup ====
+
=== Startup ===
-
''git'' clone lio-core from <code>git://git.kernel.org/pub/scm/linux/kernel/git/nab/lio-core-2.6.git</code> as follows:
+
[[targetcli]] is invoked by running ''targetcli'' as root from the command prompt of the underlying {{OS}} shell.
<pre>
<pre>
-
git clone git://git.kernel.org/pub/scm/linux/kernel/git/nab/lio-core-2.6.git
+
# targetcli
-
cd lio-core-2.6
+
Welcome to targetcli:
-
git checkout --track -b tcm_ib_srpt-38 origin/tcm_ib_srpt-38
+
-
git pull origin tcm_ib_srpt-38
+
-
cp <your_working_config_file> .config
+
-
make menuconfig
+
-
</pre>
+
-
In menuconfig:
+
Copyright (c) 2014 by Datera, Inc.
 +
All rights reserved.
-
* Select device drivers
+
Visit us at http://www.datera.io.
-
* Select [[target]] core:
+
-
<pre>
+
Using ib_srpt fabric module.
-
<M> Generic Target Core Mod (TCM) and ConfigFS Infrastructure  --->
+
Using qla2xxx fabric module.
 +
Using iscsi fabric module.
 +
Using loopback fabric module.
 +
 
 +
/> ib_srpt/ info
 +
Fabric module name: ib_srpt
 +
ConfigFS path: /sys/kernel/config/target/srpt
 +
Allowed WWN list type: free
 +
Fabric module specfile: /var/target/fabric/ib_srpt.spec
 +
Fabric module features: acls
 +
Corresponding kernel module: ib_srpt
 +
/>
</pre>
</pre>
-
* Select the modules [[FILEIO]], [[IBLOCK]], [[PSCSI]] backend drivers, and [[iSCSI]] target module.
+
Upon targetcli initialization, the underlying RTSlib loads the installed fabric modules, and creates the corresponding [[ConfigFS]] mount points (at ''/sys/kernel/config/target/<fabric>''), as specified by the associated spec files (located in ''/var/target/fabric/fabric.spec'').
-
<pre>
+
=== Display the object tree ===
-
            --- Generic Target Core Mod (TCM) and ConfigFS Infrastructure   
+
-
                  <M>  TCM/IBLOCK Subsystem Plugin for Linux/BLOCK             
+
-
                  <M>  TCM/FILEIO Subsystem Plugin for Linux/VFS         
+
-
                  <M>  TCM/pSCSI Subsystem Plugin for Linux/SCSI                                       
+
-
                  < >  TCM/STGT Subsystem Plugin                           
+
-
                  <M>  Linux-iSCSI.org iSCSI Target Mode Stack
+
-
                  [ ]  LIO-Target iscsi_debug.h ring buffer messages
+
-
                  [ ]  LIO-Target ErrorRecoveryLevel debug code                 
+
-
</pre>
+
-
* Select the Infiniband stack:
+
Use ''ls'' to list the object hierarchy, which is initially empty:
<pre>
<pre>
-
  <M> InfiniBand support  --->      
+
/> ls
 +
o- / ..................................................................... [...]
 +
  o- backstores .......................................................... [...]
 +
  | o- fileio ............................................... [0 Storage Object]
 +
  | o- iblock ............................................... [0 Storage Object]
 +
  | o- pscsi ................................................ [0 Storage Object]
 +
  | o- rd_dr ................................................ [0 Storage Object]
 +
  | o- rd_mcp ............................................... [0 Storage Object]
 +
  o- ib_srpt ........................................................ [0 Target]
 +
  o- iscsi .......................................................... [0 Target]
 +
  o- loopback ....................................................... [0 Target]
 +
  o- qla2xxx ........................................................ [0 Target]
 +
/>
</pre>
</pre>
-
*) Select the necessary IB stack support, HCA specific HW drivers and SRP Target ULP driver:
+
{{Ambox| type=info| head=Global parameter ''auto_cd_after_create''| text=After the creation of a new object, automatically enter its object context.}}
-
<pre>
+
Per default, ''auto_cd_after_create'' is set to ''true'', which automatically enters an object context (or working directory) after its creation. The examples here are modeled after this behavior.
-
                --- InfiniBand support                                                                 
+
-
                <M>  InfiniBand userspace MAD support                                                 
+
-
                <M>  InfiniBand userspace access (verbs and CM)                                      
+
-
                <M>  Mellanox HCA support                                                           
+
-
                < >  QLogic PCIe HCA support                                                         
+
-
                < >  Ammasso 1100 HCA support                                                         
+
-
                <M>  Mellanox ConnectX HCA support                                                   
+
-
                < >  NetEffect RNIC Driver                                                         
+
-
                <M>  IP-over-InfiniBand                                                               
+
-
                [ ]    IP-over-InfiniBand Connected Mode support                                     
+
-
                -*-    IP-over-InfiniBand debugging                                                   
+
-
                [ ]      IP-over-InfiniBand data path debugging                                       
+
-
                < >  InfiniBand SCSI RDMA Protocol                                                   
+
-
                <M>  InfiniBand SCSI RDMA Protocol target support                                     
+
-
                < >  iSCSI Extensions for RDMA (iSER)           
+
-
</pre>
+
Optionally, set ''auto_cd_after_create=false'' to prevent ''targetcli'' from automatically entering new object contexts after their creation:
-
 
+
-
* Save and return
+
-
 
+
-
If <code>CONFIGFS_FS=y</code> is set, a manual mount is required after the kernel boots:
+
<pre>
<pre>
-
mount -t configfs configfs /sys/kernel/config
+
/> set global auto_cd_after_create=false
 +
Parameter auto_cd_after_create is now 'false'.
 +
/>
</pre>
</pre>
-
==== Subsequent builds: git pull ====
+
=== Create a backstore ===
 +
 
 +
Create a backstore using the IBLOCK or FILEIO type devices.
-
Go to the directory where your lio-core kernel is located, and:
+
For instance, enter the top-level ''backstore'' context and create an IBLOCK backstore from a ''/dev/sdb'' block device:
<pre>
<pre>
-
cd <lio-core-2.6>
+
/> cd backstores/
-
git pull origin tcm_ib_srpt-38
+
/backstores> iblock/ create name=my_disk dev=/dev/sdb
-
make clean
+
Generating a wwn serial.
-
make oldconfig
+
Created iblock storage object my_disk using /dev/sdb.
 +
Entering new node /backstores/iblock/my_disk.
 +
/backstores/iblock/my_disk>
</pre>
</pre>
-
=== Building ===
+
''targetcli'' automatically creates a WWN serial ID for the backstore device and then changes the working context to it.
-
Kernel and modules:
+
The resulting object hierarchy looks as follows (displayed from the root object):
<pre>
<pre>
-
make
+
/> ls
-
make modules
+
o- / ..................................................................... [...]
-
make modules_install
+
  o- backstores .......................................................... [...]
-
make install
+
  | o- fileio ............................................... [0 Storage Object]
 +
  | o- iblock ............................................... [1 Storage Object]
 +
  | | o- my_disk .......................................... [/dev/sdb activated]
 +
  | o- pscsi ................................................ [0 Storage Object]
 +
  | o- rd_dr ................................................ [0 Storage Object]
 +
  | o- rd_mcp ............................................... [0 Storage Object]
 +
  o- ib_srpt ........................................................ [0 Target]
 +
  o- iscsi .......................................................... [0 Target]
 +
  o- loopback ....................................................... [0 Target]
 +
  o- qla2xxx ........................................................ [0 Target]
 +
/>
</pre>
</pre>
-
=== Installing ===
+
Alternatively, any LVM logical volume can be used as a backstore, please refer to the {{LIO Admin Manual}} on how to create them properly.
-
==== New Linux kernel ====
+
For instance, create an IBLOCK backstore on a logical volume (under ''/dev/<volume_group_name>/<logical_volume_name>''):
-
 
+
-
Use lio-core-2.6 kernel:
+
-
 
+
-
* Change <code>/boot/grub/menu.lst</code> to select your lio-core-2.6 kernel.
+
-
 
+
-
Then reboot:
+
<pre>
<pre>
-
reboot
+
/backstores> iblock/ create name=my_disk dev=/dev/vg0/lv1
 +
Generating a wwn serial.
 +
Created iblock storage object my_disk using /dev/vg0/lv1.
 +
Entering new node /backstores/iblock/my_disk.
 +
/backstores/iblock/my_disk>
</pre>
</pre>
-
and verify that the live kernel is indeed your new lio-core-2.6.git/tcm_ib_srpt-38 kernel:
+
Again, ''targetcli'' automatically creates a WWN serial ID for the backstore device and then changes the working context to it.
-
<pre>
+
{{Ambox| type=info| head=More backstore examples| text=More examples on creating backstores can be found in ''[[targetcli#Create a backstore|targetcli]]''.}}
-
uname -a  
+
-
</pre>
+
-
==== Loading LIO Target ====
+
=== Instantiate a target ===
-
Verify the presence of the following required modules with ''modprobe'': <code>ib_srpt</code>, <code>target_core_mod</code>, <code>configfs</code>, e.g.:
+
The InfiniBand ports that are available on the storage array are presented in the [[WWN]] context with the following WWPNs, for instance:
-
<pre>
+
* 0x00000000000000000002c903000e8acd
-
modprobe ib_srpt srp_max_req_size=4200
+
* 0x00000000000000000002c903000e8ace
-
</pre>
+
-
Normally all these modules should be loaded. If one is missing, you can load it as follows:
+
Instantiate an InfiniBand target, in this example for SRP over Mellanox Connect-X HCAs, on the existing IBLOCK backstore device ''my_disk'' (as set up in [[targetcli]]):
<pre>
<pre>
-
modprobe configfs
+
/backstores/iblock/my_disk> /ip_srpt create 0x00000000000000000002c903000e8acd
-
# Or do the mount command as shown above (if .config file has CONFIGFS_FS=y)
+
Created target 0x00000000000000000002c903000e8acd.
-
modprobe target_core_mod
+
Entering new node /ib_srpt/0x00000000000000000002c903000e8acd.
-
modprobe ib_srpt srp_max_req_size=4200
+
/ib_srpt/0x00...2c903000e8acd>
</pre>
</pre>
-
==== Starting LIO Target ====
+
''targetcli'' automatically changes the working context to the resulting tagged Endpoint.
-
The target will be automatically started if [[lio-utils]]/[[RTSadmin]]-frozen is installed. Otherwise, start the target manually:
+
=== Export LUNs ===
-
<pre>
+
Declare a LUN for the backstore device, to form a valid SAN storage object:
-
/etc/init.d/target start
+
-
</pre>
+
-
 
+
-
To stop target:
+
<pre>
<pre>
-
/etc/init.d/target stop
+
/ib_srpt/0x00...2c903000e8acd> luns/ create /backstores/iblock/my_disk
 +
Selected LUN 0.
 +
Successfully created LUN 0.
 +
Entering new node /ib_srpt/0x00000000000000000002c903000e8acd/luns/lun0.
 +
/ib_srpt/0x00...acd/luns/lun0>
</pre>
</pre>
-
To check the target status:
+
''targetcli'' per default automatically assigns the default ID '0' to the LUN, and then changes the working context to the SAN storage object. The target is now created, and exports ''/dev/sdb'' as LUN 0.
-
 
+
-
<pre>
+
-
/etc/init.d/target status
+
-
</pre>
+
-
 
+
-
==== User-space tools ====
+
-
 
+
-
There are two options: [[lio-utils]] and the [[RTSadmin]]-frozen packages; at the very minimum, [[lio-utils]] is required. [[RTSadmin]] requires the libreadline6 package to be installed, too. The links for these rpm packages are:
+
-
 
+
-
* http://www.risingtidesystems.com/suse-11/
+
-
* http://www.risingtidesystems.com/rhel-6-32-bit/
+
-
You can also build lio-utils.git in-house as follows.  Install ''rpmbuild'' and a ''libsnmp*-dev'' package, then:
+
Return to the underlying Endpoint as the working context, as no attributes need to be set or modified for standard LUNs:
<pre>
<pre>
-
make user_rpms
+
/ib_srpt/0x00...act/luns/lun0> cd <
 +
Taking you back to /ib_srpt/0x00000000000000000002c903000e8acd.
 +
/ib_srpt/0x00...2c903000e8acd>
</pre>
</pre>
-
The 32-bit package should be built in <code>/usr/src/packages/RPMS/</code>.
+
=== Define access rights ===
-
=== Configure from lio-utils ===
+
Configure the access rights to allow logins from initiators. This requires setting up individual access rights for each initiator, based on its WWPN.
-
Please see [[Lio-utils_HOWTO#Configuring_the_Target]] for creating a <code>/sys/kernel/config/target/core/$HBA/$DEV/</code> group using [[lio-utils#tcm_node|TCM_node]] for [[PSCSI]], [[IBLOCK]], [[FILEIO]] and [[RAMDISK]] backends.
+
Determine the WWPN for the respective InfiniBand initiator. For instance, for Linux initiator systems, use:
-
 
+
-
=== Configuration from RTSAdmin ===
+
-
 
+
-
If [[RTSadmin]] is installed:
+
<pre>
<pre>
-
rtsadmin
+
# cat /sys/class/infiniband/*/ports/*/gids/0 | sed -e s/fe80/0x0000/ -e 's/\://g'
-
  ...
+
-
<rtsadmin> create hba fileio
+
-
  ...
+
-
<rtsadmin> create storage <mylun_name> /tmp/<myfile_name> <size>
+
</pre>
</pre>
-
Where, e.g.:
+
For a simple setup, allow access to the initiator with the WWPN as determined above:
-
* ''size'' = 10G
+
-
* ''mylun_name'' could be <code>my_lun</code>
+
-
* ''myfile_name'' could be <code>my_file</code>
+
-
* This creates a storage object <code>my_lun</code> and a 10GB [[FILEIO]] backstore in <code>/tmp/my_file</code>.
+
<pre>
<pre>
-
<rtsadmin> cfspath
+
/ib_srpt/0x00...2c903000e8acd> acls/ create 0x00000000000000000002c903000e8be9
-
  Current context object lives in
+
Successfully created Node ACL for 0x00000000000000000002c903000e8be9.
-
  /sys/kernel/config/target/core/fileio_0/my_file
+
Created mapped LUN 0.
-
<rtsadmin> saveconfig
+
Entering new node /ib_srpt/0x00000000000000000002c903000e8acd/acls/
-
  ...
+
0x00000000000000000002c903000e8be9.
-
<rtsadmin> quit
+
/ib_srpt/0x00...2c903000e8be9> cd /
</pre>
</pre>
-
At this point, there are still a few manual steps required, which will soon be incorporated into ''rtsadmin'':
+
''targetcli'' per default automatically adds the appropriate mapped LUNs.
-
<pre>
+
=== Display the object tree ===
-
# Load the mlx4_ib driver and register HW as an IB HCA.
+
-
modprobe mlx4_ib
+
-
# Load the ib_srpt module, if not already done earlier
+
-
modprobe ib_srpt srp_max_req_size=4200
+
-
# Create the 'srpt/mlx4_0/mlx4_0' endpoint, and create 'lun_0' group
+
-
mkdir -p /sys/kernel/config/target/srpt/mlx4_0/mlx4_0/lun/lun_0
+
-
# Create the target core symlink from 'fileio_0/my_file/' to 'lun/lun_0/srpt_port'
+
-
ln -s /sys/kernel/config/target/core/fileio_0/my_file/ /sys/kernel/config/target/srpt/mlx4_0/mlx4_0/lun/lun_0/srpt_port
+
-
# Create the explict NodeACL for SRP Initiator:
+
-
mkdir -p /sys/kernel/config/target/srpt/mlx4_0/mlx4_0/acls/0x00000000000000000002c903000e8be9/lun_0
+
-
# Create the MappedLUN=0 for TPG LUN=0 for explict NodeACL for SRP Initiator
+
-
ln -s /sys/kernel/config/target/srpt/mlx4_0/mlx4_0/lun/lun_0 /sys/kernel/config/target/srpt/mlx4_0/mlx4_0/acls/0x00000000000000000002c903000e8be9/lun_0/mapped_lun
+
-
</pre>
+
-
Substitute the current target [[WWN]] portname (<code>mlx4_0/mlx4_0</code>) with your actual driver name from /sys/class/infiniband/, which can be obtained by:
+
The resulting InfiniBand SAN object hierarchy looks as follows (displayed from the root object):
<pre>
<pre>
-
ls /sys/class/infiniband/
+
/> ls
 +
o- / ..................................................................... [...]
 +
  o- backstores .......................................................... [...]
 +
  | o- fileio ............................................... [0 Storage Object]
 +
  | o- iblock ............................................... [1 Storage Object]
 +
  | | o- my_disk .......................................... [/dev/sdb activated]
 +
  | o- pscsi ................................................ [0 Storage Object]
 +
  | o- rd_dr ................................................ [0 Storage Object]
 +
  | o- rd_mcp ............................................... [0 Storage Object]
 +
  o- ib_srpt ........................................................ [1 Target]
 +
  | o- 0x00000000000000000002c903000e8acd ............................ [enabled]
 +
  |  o- acls .......................................................... [1 ACL]
 +
  |  | o- 0x00000000000000000002c903000e8be9 ................... [1 Mapped LUN]
 +
  |  |  o- mapped_lun0 ........................................... [lun0 (rw)]
 +
  |  o- luns .......................................................... [1 LUN]
 +
  |    o- lun0 .................................... [iblock/my_disk (/dev/sdb)]
 +
  o- iscsi .......................................................... [0 Target]
 +
  o- loopback ....................................................... [0 Target]
 +
  o- qla2xxx ........................................................ [0 Target]
 +
/>
</pre>
</pre>
-
Substitute the Initiator [[WWN]] portname (<code>0x00000000000000000002c903000e8be9)</code> with your actual IB Port GUID in long form, which can be obtained by running <code>ibstat</code> on the SRP Initiator:
+
=== Persist the configuration ===
-
<pre>
+
{{Ambox| type=warning| head=Don't forget to use ''saveconfig''!| text=Without ''saveconfig'', the {{T}} configuration will be lost upon rebooting or unloading the target service, as the target configuration will revert back to the last saved one.}}
-
truelife:/usr/src/lio-core-2.6.git# ibstat
+
-
CA 'mlx4_0'
+
-
CA type: MT26428
+
-
Number of ports: 2
+
-
Firmware version: 2.7.9230
+
-
Hardware version: b0
+
-
Node GUID: 0x0002c903000e8be8
+
-
System image GUID: 0x0002c903000e8beb
+
-
Port 1:
+
-
State: Active
+
-
Physical state: LinkUp
+
-
Rate: 40
+
-
Base lid: 1
+
-
LMC: 0
+
-
SM lid: 2
+
-
Capability mask: 0x02510868
+
-
Port GUID: 0x0002c903000e8be9
+
-
Port 2:
+
-
State: Down
+
-
Physical state: Polling
+
-
Rate: 70
+
-
Base lid: 0
+
-
LMC: 0
+
-
SM lid: 0
+
-
Capability mask: 0x02510868
+
-
Port GUID: 0x0002c903000e8bea
+
-
</pre>
+
-
Once the explict Node ACL and MappedLUN have been for the initiator configured, it's time to login from the initiator side.
+
Use ''saveconfig'' from the root context to persist the {{T}} configuration across {{OS}} reboots:
-
 
+
-
This includes using <code>ibsrpdm -c</code> to obtain the necessary string:
+
<pre>
<pre>
-
truelife:/usr/src/lio-core-2.6.git# ibsrpdm -c
+
/> saveconfig
-
id_ext=0002c903000e8acc,ioc_guid=0002c903000e8acc,dgid=fe800000000000000002c903000e8acd,pkey=ffff,service_id=0002c903000e8acc
+
WARNING: Saving rtsnode1 current configuration to disk will overwrite your boot settings.
 +
The current target configuration will become the default boot config.
 +
Are you sure? Type 'yes': yes
 +
Making backup of srpt/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
 +
Successfully updated default config /etc/target/srpt_start.sh
 +
Making backup of qla2xxx/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
 +
Successfully updated default config /etc/target/qla2xxx_start.sh
 +
Making backup of loopback/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
 +
Successfully updated default config /etc/target/loopback_start.sh
 +
Making backup of LIO-Target/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
 +
Successfully updated default config /etc/target/lio_backup-2012-02-27_23:19:37.660264.sh
 +
Making backup of Target_Core_Mod/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
 +
Successfully updated default config /etc/target/tcm_backup-2012-02-27_23:19:37.660264.sh
 +
Generated Target_Core_Mod config: /etc/target/backup/tcm_backup-2012-02-27_23:19:37.660264.sh
 +
Successfully updated default config /etc/target/lio_start.sh
 +
Successfully updated default config /etc/target/tcm_start.sh
 +
/>
</pre>
</pre>
-
And then echoing this into sysfs <code>add_target</code> attribute on the SRP Initiator to perform the SRP_LOGIN:
+
=== Spec file ===
-
<pre>
+
[[Datera]] spec files define the fabric-dependent feature set, capabilities and available target ports of the specific underlying fabric.
-
truelife:/usr/src/lio-core-2.6.git# echo -n id_ext=0002c903000e8acc,ioc_guid=0002c903000e8acc,dgid=fe800000000000000002c903000e8acd,pkey=ffff,service_id=0002c903000e8acc > /sys/class/infiniband_srp/srp-mlx4_0-1/add_target
+
-
</pre>
+
-
Notes:
+
In particular, the InfiniBand spec file ''/var/target/fabric/ib_srpt.spec'' is included via RTSlib. WWN values are extracted via ''/sys/class/infiniband/*/ports/*/gids/0'' in ''wwn_from_files_filter'' below, and are presented in the [[targetcli]] [[WWN]] context to register individual InfiniBand port GUIDs.
-
 
+
-
* By default, [[FILEIO]] uses <code>O_SYNC</code>. It allows both, exporting a shared filesystem for mounting, and creating files in the shared filesystem to be used as the backstore for raw block devices.
+
-
* Any other backstore ([[IBLOCK]], [[pSCSI]], [[RAMDISK]], etc.) can be setup accordingly:
+
-
** Use [[RTSadmin]];
+
-
** Determine the "cfspath";
+
-
** Use it as the symlink "source" when creating the [[configFS]] symlink in your <code>qla2xxx/$FC_WWPN/tpgt_$TPGT/lun/lun_$ID/<dest></code> directory.
+
-
* Any number of virtual HBAs and assoicated devices is possible. The Target subsystem plugins ([[FILEIO]], [[IBLOCK]], [[pSCSI]], [[RAMDISK]], etc.) will report the underlying HW limitiations for things like TCQ depth, <code>MaxSectors</code>, <code>TaskAbortedStatus</code>, <code>UA Interlocking</code>, etc. All of these values are available as attributes in the [[RTSadmin]] device context:
+
<pre>
<pre>
-
rtsadmin
+
# WARNING: This is a draft specfile supplied for demo purposes only.
-
  ...
+
-
fileio0/my_file(/tmp/my_file)> ls attribute
+
-
  ...
+
-
</pre>
+
-
== Results ==
+
# The ib_srpt fabric module uses the default feature set.
 +
features = acls
-
=== configfs ===
+
# The module uses hardware addresses from there
 +
wwn_from_files = /sys/class/infiniband/*/ports/*/gids/0
-
Please see [[SCSI RDMA Protocol/configFS]] for the running layout of a /sys/kernel/config/target/srpt configuration
+
# Transform 'fe80:0000:0000:0000:0002:1903:000e:8acd' WWN notation to
 +
# '0x00000000000000000002c903000e8acd'
 +
wwn_from_files_filter = "sed -e s/fe80/0x0000/ -e 's/\://g'"
-
=== lsmod output ===
+
# Non-standard module naming scheme
 +
kernel_module = ib_srpt
-
<code>
+
# The configfs group is standard
 +
configfs_group = srpt
 +
</pre>
-
  Module                  Size  Used by
+
== Scripting with RTSlib ==
-
  ib_srpt                26028  6
+
-
  ib_umad                9422  8
+
-
  iscsi_target_mod      201391  1
+
-
  mlx4_ib                38241  0
+
-
  target_core_pscsi      11045  0
+
-
  target_core_file        6307  1
+
-
  target_core_iblock      6606  1
+
-
  target_core_mod      206442  28 ib_srpt,iscsi_target_mod,target_core_pscsi,target_core_file,target_core_iblock
+
-
  configfs              19153  2 target_core_mod
+
-
  ib_iser                23172  0
+
-
  rdma_cm                23833  1 ib_iser
+
-
  ib_cm                  24878  2 ib_srpt,rdma_cm
+
-
  iw_cm                  6019  1 rdma_cm
+
-
  ib_sa                  15537  2 rdma_cm,ib_cm
+
-
  ib_mad                30330  5 ib_srpt,ib_umad,mlx4_ib,ib_cm,ib_sa
+
-
  ib_core                39716  9 ib_srpt,ib_umad,mlx4_ib,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad
+
-
  ib_addr                3982  1 rdma_cm
+
-
  ipv6                  242427  21 ib_addr
+
-
  iscsi_tcp              7729  0
+
-
  libiscsi_tcp          10485  1 iscsi_tcp
+
-
  libiscsi              29466  3 ib_iser,iscsi_tcp,libiscsi_tcp
+
-
  scsi_transport_iscsi    24624  4 ib_iser,iscsi_tcp,libiscsi
+
-
  dm_multipath          14016  0
+
-
  scsi_dh                4656  1 dm_multipath
+
-
</code>
+
=== Setup script ===
-
=== Interrupt stats ===
+
The following Python code illustrates how to setup a basic SRP target and export a mapped LUN:
-
MSI-X interrupt stats for target side operation:
+
<pre>
 +
#!/usr/bin/python
 +
# InfiniBand setup script example with RTSlib
 +
from rtslib import *
-
<code>
+
# Setup an IBLOCK backstore
 +
backstore = IBlockBackstore(3, mode='create')
 +
try:
 +
    so = IBlockStorageObject(backstore, "fioa", "/dev/fioa", gen_wwn=True)
 +
except:
 +
    backstore.delete()
 +
    raise
-
  lenny64guest0:/usr/src/lio-core-2.6.git# cat /proc/interrupts | grep mlx4
+
# Create an IB target endpoint using an ib_srpt WWPN
-
  43:  16864937  16864950  16864795  16864897  PCI-MSI-edge      mlx4-comp-0@pci:0000:00:05.0
+
fabric = FabricModule('ib_srpt')
-
  44:          0          0          0          0  PCI-MSI-edge      mlx4-comp-1@pci:0000:00:05.0
+
target = Target(fabric, '0x00000000000000000002c903000e8acd')
-
  45:          0          0          0          0  PCI-MSI-edge      mlx4-comp-2@pci:0000:00:05.0
+
tpg = TPG(target, 1)
-
  46:          0          0          0          0  PCI-MSI-edge      mlx4-comp-3@pci:0000:00:05.0
+
-
  47:      1377      1308      1338      1322  PCI-MSI-edge      mlx4-async@pci:0000:00:05.0
+
-
</code>
+
# Export LUN 0 via the 'so' StorageObject class
 +
lun0 = tpg.lun(0, so, "my_lun")
-
== Initiator ==
+
# Setup the NodeACL for an IB initiator, and create MappedLUN 0
 +
node_acl = tpg.node_acl('0x00000000000000000002c903000e8acd')
 +
mapped_lun = node_acl.mapped_lun(0, 0, False)
 +
</pre>
-
=== Kernel ring buffer ===
+
Note that while SRP [[TPG]]s are masked by [[targetcli]], they are not masked by RTSlib.
-
Initiator side ring buffer output during initiator login:
+
=== Object tree ===
-
<code>
+
The resulting object tree looks as follows:
-
   scsi7 : SRP.T10:0002C903000E8ACC
+
<pre>
-
   scsi 7:0:0:0: Direct-Access    LIO-ORG  RAMDISK-MCP      4.0 PQ: 0 ANSI: 5
+
o- / ..................................................................... [...]
-
   sd 7:0:0:0: Attached scsi generic sg2 type 0
+
   o- backstores .......................................................... [...]
-
   sd 7:0:0:0: [sdb] 524288 512-byte logical blocks: (268 MB/256 MiB)
+
   | o- fileio ............................................... [0 Storage Object]
-
  sd 7:0:0:0: [sdb] Write Protect is off
+
  | o- iblock ............................................... [1 Storage Object]
-
  sd 7:0:0:0: [sdb] Mode Sense: 2f 00 00 00
+
  | | o- my_disk .......................................... [/dev/sdb activated]
-
   sd 7:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
+
  | o- pscsi ................................................ [0 Storage Object]
-
  sdb: unknown partition table
+
   | o- rd_dr ................................................ [0 Storage Object]
-
   sd 7:0:0:0: [sdb] Attached SCSI disk
+
   | o- rd_mcp ............................................... [0 Storage Object]
 +
  o- ib_srpt ........................................................ [1 Target]
 +
    o- 0x00000000000000000002c903000e8acd ............................ [enabled]
 +
      o- acls .......................................................... [1 ACL]
 +
      | o- 0x00000000000000000002c903000e8be9 ................... [1 Mapped LUN]
 +
      |  o- mapped_lun0 ........................................... [lun0 (rw)]
 +
      o- luns .......................................................... [1 LUN]
 +
        o- lun0 .................................... [iblock/my_disk (/dev/sdb)]
 +
   o- iscsi .......................................................... [0 Target]
 +
   o- loopback ....................................................... [0 Target]
 +
  o- qla2xxx ........................................................ [0 Target]
 +
</pre>
-
</code>
+
== Specifications ==
-
=== Device identifiers ===
+
SRP was not approved as an official standard. The following specifications are available as available as [http://www.t10.org/drafts.htm T10 Working Drafts]:
-
''sg_inq -i'' initiator side output for EVPD=0x83 device identifiers:
+
* '''SCSI RDMA Protocol''' ('''SRP'''): SRP defines a SCSI protocol mapping onto the InfiniBand (tm) Architecture and/or functionally similar cluster protocols. ANSI INCITS 365-2002. Status: Final Draft. 2002-07-03
-
<code>
+
== RFCs ==
-
 
+
* {{RFC|4297|Remote Direct Memory Access (RDMA) over IP Problem Statement}}
-
  truelife:/usr/src/lio-core-2.6.git# sg_inq -i /dev/sdb
+
* {{RFC|4390|Dynamic Host Configuration Protocol (DHCP) over InfiniBand}}
-
  VPD INQUIRY: Device Identification page
+
* {{RFC|4391|Transmission of IP over InfiniBand (IPoIB)}}
-
    Designation descriptor number 1, descriptor length: 20
+
* {{RFC|4392|IP over InfiniBand (IPoIB) Architecture}}
-
      designator_type: NAA,  code_set: Binary
+
* {{RFC|4755|IP over InfiniBand: Connected Mode}}
-
      associated with the addressed logical unit
+
* {{RFC|5040|A Remote Direct Memory Access Protocol Specification}}
-
        NAA 6, IEEE Company_id: 0x1405
+
* {{RFC|5045|Applicability of Remote Direct Memory Access Protocol (RDMA) and Direct Data Placement Protocol (DDP)}}
-
        Vendor Specific Identifier: 0x929cc061d
+
* {{RFC|5046|Internet Small Computer System Interface (iSCSI) Extensions for Remote Direct Memory Access (RDMA)}}
-
        Vendor Specific Identifier Extension: 0xe082d4d98d810cdc
+
* {{RFC|5047|DA: Datamover Architecture for the Internet Small Computer System Interface (iSCSI)}}
-
        [0x6001405929cc061de082d4d98d810cdc]
+
-
    Designation descriptor number 2, descriptor length: 61
+
-
      designator_type: T10 vendor identification,  code_set: ASCII
+
-
      associated with the addressed logical unit
+
-
        vendor id: LIO-ORG
+
-
        vendor specific: RAMDISK-MCP:929cc061-e082-4d98-810c-c90b421bd066
+
-
    Designation descriptor number 3, descriptor length: 8
+
-
      transport: Remote Direct Memory Access (RDMA)
+
-
      designator_type: Relative target port,  code_set: Binary
+
-
      associated with the target port
+
-
        Relative target port: 0x2
+
-
    Designation descriptor number 4, descriptor length: 8
+
-
      transport: Remote Direct Memory Access (RDMA)
+
-
      designator_type: Target port group,  code_set: Binary
+
-
      associated with the target port
+
-
        Target port group: 0x0
+
-
    Designation descriptor number 5, descriptor length: 8
+
-
      designator_type: Logical unit group,  code_set: Binary
+
-
      associated with the addressed logical unit
+
-
        Logical unit group: 0x0
+
-
    Designation descriptor number 6, descriptor length: 32
+
-
      transport: Remote Direct Memory Access (RDMA)
+
-
      designator_type: SCSI name string,  code_set: UTF-8
+
-
      associated with the target port
+
-
        SCSI name string:
+
-
        0x2c903000e8acc,t,0x0001
+
-
 
+
-
</code>
+
-
 
+
-
== ibsrpdm output ==
+
-
 
+
-
<code>
+
-
 
+
-
  truelife:/usr/src/lio-core-2.6.git# ibsrpdm
+
-
  IO Unit Info:
+
-
      port LID:        0002
+
-
      port GID:        fe800000000000000002c903000e8acd
+
-
      change ID:      0001
+
-
      max controllers: 0x10
+
-
      controller[  1]
+
-
          GUID:      0002c903000e8acc
+
-
          vendor ID: 000002
+
-
          device ID: 00673c
+
-
          IO class : 0100
+
-
          ID:        Linux SRP target
+
-
          service entries: 1
+
-
              service[  0]: 0002c903000e8acc / SRP.T10:0002c903000e8acc
+
-
 
+
-
</code>
+
-
 
+
-
== Timeline ==
+
-
{{LIO Timeline}}
+
== See also ==
== See also ==
-
* [[ConfigFS]] low level kernel API: SRP/configFS
+
* [[{{OS}}]]
-
* [[InfiniBand]], [[Target]] / [[SCSI]]
+
* {{Target}}, [[targetcli]]
-
* Fabric modules: [[iSCSI]], [[Fibre Channel]], [[Fibre Channel over Ethernet|FCoE]], [[tcm_loop]]
+
* [[FCoE]], [[Fibre Channel]], [[iSCSI]], [[iSER]], [[tcm_loop]], [[vHost]]
== Notes ==
== Notes ==
{{Reflist}}
{{Reflist}}
 +
 +
== Wikipedia entries ==
 +
* [http://en.wikipedia.org/wiki/InfiniBand InfiniBand] (IB)
 +
* [http://en.wikipedia.org/wiki/IWARP Internet Wide Area RDMA Protocol] (iWARP)
 +
* [http://en.wikipedia.org/wiki/Remote_direct_memory_access Remote direct memory access] (RDMA)
 +
* [http://en.wikipedia.org/wiki/SCSI_RDMA_Protocol SCSI RDMA Protocol] (SRP)
 +
* [http://en.wikipedia.org/wiki/Virtual_Interface_Architecture Virtual Interface Architecture] (VIA)
== External links ==
== External links ==
 +
* {{LIO Admin Manual}}
 +
* RTSlib Reference Guide {{Lib Ref Guide HTML}}{{Lib Ref Guide PDF}}
 +
* [http://en.wikipedia.org/wiki/SCSI_RDMA_Protocol SRP] Wikipedia entry
 +
* {{cite web| url=http://www.youtube.com/watch?v=4UVRuq-lgKo| title=SRP Update and Directions| date=2012-05-27| publisher=openfabrics.org}}
* [http://marc.info/?l=linux-rdma&r=1&w=2 linux-rdma] mailing list
* [http://marc.info/?l=linux-rdma&r=1&w=2 linux-rdma] mailing list
-
* [http://en.wikipedia.org/wiki/SCSI_RDMA_Protocol SRP] Wikipedia entry
+
* [http://www.openfabrics.org/ OpenFabrics Alliance]
-
* [http://en.wikipedia.org/wiki/Infiniband InfiniBand] Wikipedia entry
+
* [http://www.mellanox.com/ Mellanox] website
* [http://www.mellanox.com/ Mellanox] website
* [http://www.t10.org/index.html T10] home page
* [http://www.t10.org/index.html T10] home page
 +
 +
{{LIO Timeline}}
[[Category:Fabric modules]]
[[Category:Fabric modules]]
[[Category:InfiniBand]]
[[Category:InfiniBand]]
[[Category:Network protocols]]
[[Category:Network protocols]]

Latest revision as of 02:42, 7 August 2015

LinuxIO
Logo
LIO 150513.png
InfiniBand SRP fabric module
Original author(s) Vu Pham
Bart Van Assche
Nicholas Bellinger
Developer(s) Mellanox Technologies, Ltd.
Datera, Inc.
Initial release March 18, 2012 (2012-03-18)
Stable release 4.1.0 / June 20, 2012;
7 years ago
 (2012-06-20)
Preview release 4.2.0-rc5 / June 28, 2012;
7 years ago
 (2012-06-28)
Development status Production
Written in C
Operating system Linux
Type Fabric module
License GNU General Public License, version 2 (GPLv2)
Website datera.io
See LIO for a complete overview over all fabric modules.

The SCSI RDMA Protocol (SRP) is a network protocol that allows one computer system to access SCSI devices attached to another computer system via RDMA.[1][2]

Contents

Overview

SRP was designed to use RDMA networks efficiently. RDMA allows lower latencies and higher throughput than TCP/IP protocols, but requires network adapters with native RDMA support, e.g., HCAs for InfiniBand, RNICs on "lossless" (DCB) Ethernet, or TOE NICs with iWARP for standard Ethernet.

SRP is based on the SCSI protocol, which is a point-to-point protocol with corresponding design limitations. In contrast, iSER is based on iSCSI, and thus better accommodates modern network requirements, including complex topologies, multipathing, target discovery, etc. Hence, iSER is most likely the best choice for InfiniBand networks going forward.

SRP never became an official standard: the latest draft of the SRP protocol, revision 16a, dates from July 3, 2002.[1]

The InfiniBand/SRP fabric module (srpt.ko, Linux kernel driver database) for the LinuxIO was released with Linux kernel 3.3 on March 18, 2012 (2012-03-18).[3]

targetcli

targetcli from Datera, Inc. is used to configure InfiniBand targets. targetcli aggregates LIO service modules via a core library, and exports them through an API, to provide a unified single-node SAN configuration shell, independently of the underlying fabric(s).

Cheat sheet

Command Comment
/backstores/iblock create my_disk /dev/sdb Create the LUN my_disk on the block device /dev/sdb
/ib_srpt create <WWPN> Create an SRP target
In /ib_srpt/<WWPN>:
luns/ create /backstores/iblock/my_disk
Export the LUN my_disk
In /ib_srpt/<WWPN>:
acls/ create <Initiator WWPN>
Allow access for the initiator at <WWPN>
/saveconfig Commit the configuration

Startup

targetcli is invoked by running targetcli as root from the command prompt of the underlying LIO shell.

# targetcli
Welcome to targetcli:

 Copyright (c) 2014 by Datera, Inc.
 All rights reserved.

Visit us at http://www.datera.io.

Using ib_srpt fabric module.
Using qla2xxx fabric module.
Using iscsi fabric module.
Using loopback fabric module.

/> ib_srpt/ info
Fabric module name: ib_srpt
ConfigFS path: /sys/kernel/config/target/srpt
Allowed WWN list type: free
Fabric module specfile: /var/target/fabric/ib_srpt.spec
Fabric module features: acls
Corresponding kernel module: ib_srpt
/>

Upon targetcli initialization, the underlying RTSlib loads the installed fabric modules, and creates the corresponding ConfigFS mount points (at /sys/kernel/config/target/<fabric>), as specified by the associated spec files (located in /var/target/fabric/fabric.spec).

Display the object tree

Use ls to list the object hierarchy, which is initially empty:

/> ls
o- / ..................................................................... [...]
  o- backstores .......................................................... [...]
  | o- fileio ............................................... [0 Storage Object]
  | o- iblock ............................................... [0 Storage Object]
  | o- pscsi ................................................ [0 Storage Object]
  | o- rd_dr ................................................ [0 Storage Object]
  | o- rd_mcp ............................................... [0 Storage Object]
  o- ib_srpt ........................................................ [0 Target]
  o- iscsi .......................................................... [0 Target]
  o- loopback ....................................................... [0 Target]
  o- qla2xxx ........................................................ [0 Target]
/>

Per default, auto_cd_after_create is set to true, which automatically enters an object context (or working directory) after its creation. The examples here are modeled after this behavior.

Optionally, set auto_cd_after_create=false to prevent targetcli from automatically entering new object contexts after their creation:

/> set global auto_cd_after_create=false
Parameter auto_cd_after_create is now 'false'.
/>

Create a backstore

Create a backstore using the IBLOCK or FILEIO type devices.

For instance, enter the top-level backstore context and create an IBLOCK backstore from a /dev/sdb block device:

/> cd backstores/
/backstores> iblock/ create name=my_disk dev=/dev/sdb
Generating a wwn serial.
Created iblock storage object my_disk using /dev/sdb.
Entering new node /backstores/iblock/my_disk.
/backstores/iblock/my_disk>

targetcli automatically creates a WWN serial ID for the backstore device and then changes the working context to it.

The resulting object hierarchy looks as follows (displayed from the root object):

/> ls
o- / ..................................................................... [...]
  o- backstores .......................................................... [...]
  | o- fileio ............................................... [0 Storage Object]
  | o- iblock ............................................... [1 Storage Object]
  | | o- my_disk .......................................... [/dev/sdb activated]
  | o- pscsi ................................................ [0 Storage Object]
  | o- rd_dr ................................................ [0 Storage Object]
  | o- rd_mcp ............................................... [0 Storage Object]
  o- ib_srpt ........................................................ [0 Target]
  o- iscsi .......................................................... [0 Target]
  o- loopback ....................................................... [0 Target]
  o- qla2xxx ........................................................ [0 Target]
/>

Alternatively, any LVM logical volume can be used as a backstore, please refer to the LIO Admin Manual on how to create them properly.

For instance, create an IBLOCK backstore on a logical volume (under /dev/<volume_group_name>/<logical_volume_name>):

/backstores> iblock/ create name=my_disk dev=/dev/vg0/lv1
Generating a wwn serial.
Created iblock storage object my_disk using /dev/vg0/lv1.
Entering new node /backstores/iblock/my_disk.
/backstores/iblock/my_disk>

Again, targetcli automatically creates a WWN serial ID for the backstore device and then changes the working context to it.

Instantiate a target

The InfiniBand ports that are available on the storage array are presented in the WWN context with the following WWPNs, for instance:

Instantiate an InfiniBand target, in this example for SRP over Mellanox Connect-X HCAs, on the existing IBLOCK backstore device my_disk (as set up in targetcli):

/backstores/iblock/my_disk> /ip_srpt create 0x00000000000000000002c903000e8acd
Created target 0x00000000000000000002c903000e8acd.
Entering new node /ib_srpt/0x00000000000000000002c903000e8acd.
/ib_srpt/0x00...2c903000e8acd>

targetcli automatically changes the working context to the resulting tagged Endpoint.

Export LUNs

Declare a LUN for the backstore device, to form a valid SAN storage object:

/ib_srpt/0x00...2c903000e8acd> luns/ create /backstores/iblock/my_disk
Selected LUN 0.
Successfully created LUN 0.
Entering new node /ib_srpt/0x00000000000000000002c903000e8acd/luns/lun0.
/ib_srpt/0x00...acd/luns/lun0>

targetcli per default automatically assigns the default ID '0' to the LUN, and then changes the working context to the SAN storage object. The target is now created, and exports /dev/sdb as LUN 0.

Return to the underlying Endpoint as the working context, as no attributes need to be set or modified for standard LUNs:

/ib_srpt/0x00...act/luns/lun0> cd <
Taking you back to /ib_srpt/0x00000000000000000002c903000e8acd.
/ib_srpt/0x00...2c903000e8acd>

Define access rights

Configure the access rights to allow logins from initiators. This requires setting up individual access rights for each initiator, based on its WWPN.

Determine the WWPN for the respective InfiniBand initiator. For instance, for Linux initiator systems, use:

# cat /sys/class/infiniband/*/ports/*/gids/0 | sed -e s/fe80/0x0000/ -e 's/\://g'

For a simple setup, allow access to the initiator with the WWPN as determined above:

/ib_srpt/0x00...2c903000e8acd> acls/ create 0x00000000000000000002c903000e8be9
Successfully created Node ACL for 0x00000000000000000002c903000e8be9.
Created mapped LUN 0.
Entering new node /ib_srpt/0x00000000000000000002c903000e8acd/acls/
0x00000000000000000002c903000e8be9.
/ib_srpt/0x00...2c903000e8be9> cd /

targetcli per default automatically adds the appropriate mapped LUNs.

Display the object tree

The resulting InfiniBand SAN object hierarchy looks as follows (displayed from the root object):

/> ls
o- / ..................................................................... [...]
  o- backstores .......................................................... [...]
  | o- fileio ............................................... [0 Storage Object]
  | o- iblock ............................................... [1 Storage Object]
  | | o- my_disk .......................................... [/dev/sdb activated]
  | o- pscsi ................................................ [0 Storage Object]
  | o- rd_dr ................................................ [0 Storage Object]
  | o- rd_mcp ............................................... [0 Storage Object]
  o- ib_srpt ........................................................ [1 Target]
  | o- 0x00000000000000000002c903000e8acd ............................ [enabled]
  |   o- acls .......................................................... [1 ACL]
  |   | o- 0x00000000000000000002c903000e8be9 ................... [1 Mapped LUN]
  |   |   o- mapped_lun0 ........................................... [lun0 (rw)]
  |   o- luns .......................................................... [1 LUN]
  |     o- lun0 .................................... [iblock/my_disk (/dev/sdb)]
  o- iscsi .......................................................... [0 Target]
  o- loopback ....................................................... [0 Target]
  o- qla2xxx ........................................................ [0 Target]
/>

Persist the configuration

Use saveconfig from the root context to persist the LIO configuration across LIO reboots:

/> saveconfig
WARNING: Saving rtsnode1 current configuration to disk will overwrite your boot settings.
The current target configuration will become the default boot config.
Are you sure? Type 'yes': yes
Making backup of srpt/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
Successfully updated default config /etc/target/srpt_start.sh
Making backup of qla2xxx/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
Successfully updated default config /etc/target/qla2xxx_start.sh
Making backup of loopback/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
Successfully updated default config /etc/target/loopback_start.sh
Making backup of LIO-Target/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
Successfully updated default config /etc/target/lio_backup-2012-02-27_23:19:37.660264.sh
Making backup of Target_Core_Mod/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
Successfully updated default config /etc/target/tcm_backup-2012-02-27_23:19:37.660264.sh
Generated Target_Core_Mod config: /etc/target/backup/tcm_backup-2012-02-27_23:19:37.660264.sh
Successfully updated default config /etc/target/lio_start.sh
Successfully updated default config /etc/target/tcm_start.sh
/>

Spec file

Datera spec files define the fabric-dependent feature set, capabilities and available target ports of the specific underlying fabric.

In particular, the InfiniBand spec file /var/target/fabric/ib_srpt.spec is included via RTSlib. WWN values are extracted via /sys/class/infiniband/*/ports/*/gids/0 in wwn_from_files_filter below, and are presented in the targetcli WWN context to register individual InfiniBand port GUIDs.

# WARNING: This is a draft specfile supplied for demo purposes only.

# The ib_srpt fabric module uses the default feature set.
features = acls

# The module uses hardware addresses from there
wwn_from_files = /sys/class/infiniband/*/ports/*/gids/0

# Transform 'fe80:0000:0000:0000:0002:1903:000e:8acd' WWN notation to
# '0x00000000000000000002c903000e8acd'
wwn_from_files_filter = "sed -e s/fe80/0x0000/ -e 's/\://g'"

# Non-standard module naming scheme
kernel_module = ib_srpt

# The configfs group is standard
configfs_group = srpt

Scripting with RTSlib

Setup script

The following Python code illustrates how to setup a basic SRP target and export a mapped LUN:

#!/usr/bin/python
# InfiniBand setup script example with RTSlib
from rtslib import *

# Setup an IBLOCK backstore
backstore = IBlockBackstore(3, mode='create')
try:
    so = IBlockStorageObject(backstore, "fioa", "/dev/fioa", gen_wwn=True)
except:
    backstore.delete()
    raise

# Create an IB target endpoint using an ib_srpt WWPN
fabric = FabricModule('ib_srpt')
target = Target(fabric, '0x00000000000000000002c903000e8acd')
tpg = TPG(target, 1)

# Export LUN 0 via the 'so' StorageObject class
lun0 = tpg.lun(0, so, "my_lun")

# Setup the NodeACL for an IB initiator, and create MappedLUN 0
node_acl = tpg.node_acl('0x00000000000000000002c903000e8acd')
mapped_lun = node_acl.mapped_lun(0, 0, False)

Note that while SRP TPGs are masked by targetcli, they are not masked by RTSlib.

Object tree

The resulting object tree looks as follows:

o- / ..................................................................... [...]
  o- backstores .......................................................... [...]
  | o- fileio ............................................... [0 Storage Object]
  | o- iblock ............................................... [1 Storage Object]
  | | o- my_disk .......................................... [/dev/sdb activated]
  | o- pscsi ................................................ [0 Storage Object]
  | o- rd_dr ................................................ [0 Storage Object]
  | o- rd_mcp ............................................... [0 Storage Object]
  o- ib_srpt ........................................................ [1 Target]
    o- 0x00000000000000000002c903000e8acd ............................ [enabled]
      o- acls .......................................................... [1 ACL]
      | o- 0x00000000000000000002c903000e8be9 ................... [1 Mapped LUN]
      |   o- mapped_lun0 ........................................... [lun0 (rw)]
      o- luns .......................................................... [1 LUN]
        o- lun0 .................................... [iblock/my_disk (/dev/sdb)]
  o- iscsi .......................................................... [0 Target]
  o- loopback ....................................................... [0 Target]
  o- qla2xxx ........................................................ [0 Target]

Specifications

SRP was not approved as an official standard. The following specifications are available as available as T10 Working Drafts:

RFCs

See also

Notes

  1. a b ANSI T10 SRPr16a, www.t10.org.
  2. ANSI T10 SRPr16a, web.archive.org
  3. Linus Torvalds (2012-03-18). "Linux 3.3". lkml.org. 

Wikipedia entries

External links

Timeline of the LinuxIO
Release Details 2011 2012 2013 2014 2015
123456789101112 123456789101112 123456789101112 123456789101112 123456789101112
4.x Version 4.0 4.1
Feature LIO Core Loop back FCoE iSCSI Perf SRP
CM WQ FC
USB
1394
vHost Perf Misc 16 GFC iSER Misc VAAI Misc DIF Core
NPIV
DIF iSER DIF FC vhost TCMU Xen Misc Misc virtio 1.0 Misc NVMe OF
Linux 2.6.38 2.6.39 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18 3.19 3.20 3.21 3.22
Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox
Google AdSense