SCSI RDMA Protocol

The Linux SCSI Target Wiki

(Difference between revisions)
Jump to: navigation, search
m
m
Line 39: Line 39:
Use ''[[targetcli]]'' from {{RTS full}} to configure SRP targets, as described in [[InfiniBand]].
Use ''[[targetcli]]'' from {{RTS full}} to configure SRP targets, as described in [[InfiniBand]].
-
 
-
== Setup ==
 
-
 
-
=== Getting the sources ===
 
-
 
-
==== Initial build: git setup ====
 
-
 
-
''git'' clone lio-core from <code>git://git.kernel.org/pub/scm/linux/kernel/git/nab/lio-core-2.6.git</code> as follows:
 
-
 
-
<pre>
 
-
git clone git://git.kernel.org/pub/scm/linux/kernel/git/nab/lio-core-2.6.git
 
-
cd lio-core-2.6
 
-
git checkout --track -b tcm_ib_srpt-38 origin/tcm_ib_srpt-38
 
-
git pull origin tcm_ib_srpt-38
 
-
cp <your_working_config_file> .config
 
-
make menuconfig
 
-
</pre>
 
-
 
-
In menuconfig:
 
-
 
-
* Select device drivers
 
-
* Select [[target]] core:
 
-
 
-
<pre>
 
-
<M> Generic Target Core Mod (TCM) and ConfigFS Infrastructure  --->
 
-
</pre>
 
-
 
-
* Select the modules [[FILEIO]], [[IBLOCK]], [[PSCSI]] backend drivers, and [[iSCSI]] target module.
 
-
 
-
<pre>
 
-
            --- Generic Target Core Mod (TCM) and ConfigFS Infrastructure   
 
-
                  <M>  TCM/IBLOCK Subsystem Plugin for Linux/BLOCK             
 
-
                  <M>  TCM/FILEIO Subsystem Plugin for Linux/VFS         
 
-
                  <M>  TCM/pSCSI Subsystem Plugin for Linux/SCSI                                       
 
-
                  < >  TCM/STGT Subsystem Plugin                           
 
-
                  <M>  Linux-iSCSI.org iSCSI Target Mode Stack
 
-
                  [ ]  LIO-Target iscsi_debug.h ring buffer messages
 
-
                  [ ]  LIO-Target ErrorRecoveryLevel debug code                 
 
-
</pre>
 
-
 
-
* Select the Infiniband stack:
 
-
 
-
<pre>
 
-
<M> InfiniBand support  --->     
 
-
</pre>
 
-
 
-
*) Select the necessary IB stack support, HCA specific HW drivers and SRP Target ULP driver:
 
-
 
-
<pre>
 
-
                --- InfiniBand support                                                                 
 
-
                <M>  InfiniBand userspace MAD support                                                 
 
-
                <M>  InfiniBand userspace access (verbs and CM)                                     
 
-
                <M>  Mellanox HCA support                                                           
 
-
                < >  QLogic PCIe HCA support                                                         
 
-
                < >  Ammasso 1100 HCA support                                                         
 
-
                <M>  Mellanox ConnectX HCA support                                                   
 
-
                < >  NetEffect RNIC Driver                                                         
 
-
                <M>  IP-over-InfiniBand                                                               
 
-
                [ ]    IP-over-InfiniBand Connected Mode support                                     
 
-
                -*-    IP-over-InfiniBand debugging                                                   
 
-
                [ ]      IP-over-InfiniBand data path debugging                                       
 
-
                < >  InfiniBand SCSI RDMA Protocol                                                   
 
-
                <M>  InfiniBand SCSI RDMA Protocol target support                                     
 
-
                < >  iSCSI Extensions for RDMA (iSER)           
 
-
 
-
</pre>
 
-
 
-
* Save and return
 
-
 
-
If <code>CONFIGFS_FS=y</code> is set, a manual mount is required after the kernel boots:
 
-
 
-
<pre>
 
-
mount -t configfs configfs /sys/kernel/config
 
-
</pre>
 
-
 
-
==== Subsequent builds: git pull ====
 
-
 
-
Go to the directory where your lio-core kernel is located, and:
 
-
 
-
<pre>
 
-
cd <lio-core-2.6>
 
-
git pull origin tcm_ib_srpt-38
 
-
make clean
 
-
make oldconfig
 
-
</pre>
 
-
 
-
=== Building ===
 
-
 
-
Kernel and modules:
 
-
 
-
<pre>
 
-
make
 
-
make modules
 
-
make modules_install
 
-
make install
 
-
</pre>
 
-
 
-
==== Installing ====
 
-
 
-
==== New Linux kernel ====
 
-
 
-
Use lio-core-2.6 kernel:
 
-
 
-
* Change <code>/boot/grub/menu.lst</code> to select your lio-core-2.6 kernel.
 
-
 
-
Then reboot:
 
-
 
-
<pre>
 
-
reboot
 
-
</pre>
 
-
 
-
and verify that the live kernel is indeed your new lio-core-2.6.git/tcm_ib_srpt-38 kernel:
 
-
 
-
<pre>
 
-
uname -a
 
-
</pre>
 
-
 
-
==== Loading LIO Target ====
 
-
 
-
Verify the presence of the following required modules with ''modprobe'': <code>ib_srpt</code>, <code>target_core_mod</code>, <code>configfs</code>, e.g.:
 
-
 
-
<pre>
 
-
modprobe ib_srpt srp_max_req_size=4200
 
-
</pre>
 
-
 
-
Normally all these modules should be loaded. If one is missing, you can load it as follows:
 
-
 
-
<pre>
 
-
modprobe configfs
 
-
# Or do the mount command as shown above (if .config file has CONFIGFS_FS=y)
 
-
modprobe target_core_mod
 
-
modprobe ib_srpt srp_max_req_size=4200
 
-
</pre>
 
-
 
-
==== Starting LIO Target ====
 
-
 
-
The target will be automatically started if [[targetcli]]-frozen is installed. Otherwise, start the target manually:
 
-
 
-
<pre>
 
-
/etc/init.d/target start
 
-
</pre>
 
-
 
-
To stop target:
 
-
 
-
<pre>
 
-
/etc/init.d/target stop
 
-
</pre>
 
-
 
-
To check the target status:
 
-
 
-
<pre>
 
-
/etc/init.d/target status
 
-
</pre>
 
-
 
-
=== Results ===
 
-
 
-
==== configfs ====
 
-
 
-
Please see [[SCSI RDMA Protocol/configFS]] for the running layout of a /sys/kernel/config/target/srpt configuration.
 
-
 
-
==== lsmod output ====
 
-
 
-
<pre>
 
-
Module                  Size  Used by
 
-
ib_srpt                26028  6
 
-
ib_umad                9422  8
 
-
iscsi_target_mod      201391  1
 
-
mlx4_ib                38241  0
 
-
target_core_pscsi      11045  0
 
-
target_core_file        6307  1
 
-
target_core_iblock      6606  1
 
-
target_core_mod      206442  28 ib_srpt,iscsi_target_mod,target_core_pscsi,target_core_file,target_core_iblock
 
-
configfs              19153  2 target_core_mod
 
-
ib_iser                23172  0
 
-
rdma_cm                23833  1 ib_iser
 
-
ib_cm                  24878  2 ib_srpt,rdma_cm
 
-
iw_cm                  6019  1 rdma_cm
 
-
ib_sa                  15537  2 rdma_cm,ib_cm
 
-
ib_mad                30330  5 ib_srpt,ib_umad,mlx4_ib,ib_cm,ib_sa
 
-
ib_core                39716  9 ib_srpt,ib_umad,mlx4_ib,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad
 
-
ib_addr                3982  1 rdma_cm
 
-
ipv6                  242427  21 ib_addr
 
-
iscsi_tcp              7729  0
 
-
libiscsi_tcp          10485  1 iscsi_tcp
 
-
libiscsi              29466  3 ib_iser,iscsi_tcp,libiscsi_tcp
 
-
scsi_transport_iscsi    24624  4 ib_iser,iscsi_tcp,libiscsi
 
-
dm_multipath          14016  0
 
-
scsi_dh                4656  1 dm_multipath
 
-
</pre>
 
-
 
-
==== Interrupt stats ====
 
-
 
-
MSI-X interrupt stats for target side operation:
 
-
 
-
<pre>
 
-
lenny64guest0:/usr/src/lio-core-2.6.git# cat /proc/interrupts | grep mlx4
 
-
43:  16864937  16864950  16864795  16864897  PCI-MSI-edge      mlx4-comp-0@pci:0000:00:05.0
 
-
44:          0          0          0          0  PCI-MSI-edge      mlx4-comp-1@pci:0000:00:05.0
 
-
45:          0          0          0          0  PCI-MSI-edge      mlx4-comp-2@pci:0000:00:05.0
 
-
46:          0          0          0          0  PCI-MSI-edge      mlx4-comp-3@pci:0000:00:05.0
 
-
47:      1377      1308      1338      1322  PCI-MSI-edge      mlx4-async@pci:0000:00:05.0
 
-
</pre>
 
-
 
-
==== ibsrpdm output ====
 
-
 
-
<pre>
 
-
# ibsrpdm
 
-
IO Unit Info:
 
-
    port LID:        0002
 
-
    port GID:        fe800000000000000002c903000e8acd
 
-
    change ID:      0001
 
-
    max controllers: 0x10
 
-
    controller[  1]
 
-
        GUID:      0002c903000e8acc
 
-
        vendor ID: 000002
 
-
        device ID: 00673c
 
-
        IO class : 0100
 
-
        ID:        Linux SRP target
 
-
        service entries: 1
 
-
        service[  0]: 0002c903000e8acc / SRP.T10:0002c903000e8acc
 
-
</pre>
 
== Initiator ==
== Initiator ==

Revision as of 21:16, 4 July 2012

LIO Target
Logo
LIO 150513.png
Fibre Channel fabric module(s)
Original author(s) Vu Pham
Bart Van Assche
Nicholas Bellinger
Developer(s) Mellanox Technologies, Ltd.
Initial release March 18, 2012 (2012-03-18)
Stable release 4.1.0 / June 20, 2012;
7 years ago
 (2012-06-20)
Preview release 4.2.0-rc5 / June 28, 2012;
7 years ago
 (2012-06-28)
Development status Production
Written in C
Operating system Linux
Type Fabric module
License GNU General Public License
Website www.mellanox.com
See Target for a complete overview over all fabric modules.

An SCSI RDMA Protocol (SRP, aka SCSI Remote Protocol) fabric module (ib_srpt) for Mellanox HCAs is supported.

Contents

Overview

SRP is based on the The SCSI RDMA Protocol. RDMA defines a SCSI mapping onto the Infiniband architecture and/or functionally similar cluster protocols, and generally allows higher throughput and lower latency than TCP/IP based communication. RDMA is only possible with network adapters that have hardware support for RDMA. Examples of such network adapters are InfiniBand HCAs and 10 GbE network adapters with iWARP support. While the SRP protocol has been designed to use RDMA networks efficiently, it is also possible to implement the SRP protocol over networks that do not support RDMA.

SRP is easier to implement than iSER, but iSER offers more management functionality, e.g. the target discovery infrastructure enabled by the iSCSI protocol.

The InfiniBand/SRP fabric module (ib_srpt.ko) went upstream into the Linux 3.3 kernel on 1/18/2012.[1]

Configuration

Use targetcli from Datera, Inc. to configure SRP targets, as described in InfiniBand.

Initiator

Kernel ring buffer

Initiator side ring buffer output during initiator login:

scsi7 : SRP.T10:0002C903000E8ACC
scsi 7:0:0:0: Direct-Access     LIO-ORG  RAMDISK-MCP      4.0  PQ: 0 ANSI: 5
sd 7:0:0:0: Attached scsi generic sg2 type 0
sd 7:0:0:0: [sdb] 524288 512-byte logical blocks: (268 MB/256 MiB)
sd 7:0:0:0: [sdb] Write Protect is off
sd 7:0:0:0: [sdb] Mode Sense: 2f 00 00 00
sd 7:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
 sdb: unknown partition table
sd 7:0:0:0: [sdb] Attached SCSI disk

Device identifiers

sg_inq -i initiator side output for EVPD=0x83 device identifiers:

# sg_inq -i /dev/sdb
VPD INQUIRY: Device Identification page
  Designation descriptor number 1, descriptor length: 20
    designator_type: NAA,  code_set: Binary
    associated with the addressed logical unit
      NAA 6, IEEE Company_id: 0x1405
      Vendor Specific Identifier: 0x929cc061d
      Vendor Specific Identifier Extension: 0xe082d4d98d810cdc
      [0x6001405929cc061de082d4d98d810cdc]
  Designation descriptor number 2, descriptor length: 61
    designator_type: T10 vendor identification,  code_set: ASCII
    associated with the addressed logical unit
      vendor id: LIO-ORG
      vendor specific: RAMDISK-MCP:929cc061-e082-4d98-810c-c90b421bd066
  Designation descriptor number 3, descriptor length: 8
    transport: Remote Direct Memory Access (RDMA)
    designator_type: Relative target port,  code_set: Binary
    associated with the target port
      Relative target port: 0x2
  Designation descriptor number 4, descriptor length: 8
    transport: Remote Direct Memory Access (RDMA)
    designator_type: Target port group,  code_set: Binary
    associated with the target port
      Target port group: 0x0
  Designation descriptor number 5, descriptor length: 8
    designator_type: Logical unit group,  code_set: Binary
    associated with the addressed logical unit
      Logical unit group: 0x0
  Designation descriptor number 6, descriptor length: 32
    transport: Remote Direct Memory Access (RDMA)
    designator_type: SCSI name string,  code_set: UTF-8
    associated with the target port
      SCSI name string:
      0x2c903000e8acc,t,0x0001

Specifications

SRP was not approved as an official standard. The following specifications are available as available as T10 Working Drafts:

Timeline

Timeline of the LinuxIO
Release Details 2011 2012 2013 2014 2015
123456789101112 123456789101112 123456789101112 123456789101112 123456789101112
4.x Version 4.0 4.1
Feature LIO Core Loop back FCoE iSCSI Perf SRP
CM WQ FC
USB
1394
vHost Perf Misc 16 GFC iSER Misc VAAI Misc DIF Core
NPIV
DIF iSER DIF FC vhost TCMU Xen Misc Misc virtio 1.0 Misc NVMe OF
Linux 2.6.38 2.6.39 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18 3.19 3.20 3.21 3.22

See also

Notes

  1. Linus Torvalds (1/18/2012). "InfiniBand/SRP merge". lkml.org. 

External links

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox
Google AdSense