SCSI RDMA Protocol

The Linux SCSI Target Wiki

Revision as of 18:20, 4 July 2012 by Marcf (Talk | contribs)
Jump to: navigation, search
LIO Target
Logo
LIO 150513.png
Fibre Channel fabric module(s)
Original author(s) Vu Pham
Bart Van Assche
Nicholas Bellinger
Developer(s) Mellanox Technologies, Ltd.
Initial release March 18, 2012 (2012-03-18)
Stable release 4.1.0 / June 20, 2012;
7 years ago
 (2012-06-20)
Preview release 4.2.0-rc5 / June 28, 2012;
7 years ago
 (2012-06-28)
Development status Production
Written in C
Operating system Linux
Type Fabric module
License GNU General Public License
Website www.mellanox.com
See Target for a complete overview over all fabric modules.

An SCSI RDMA Protocol (SRP, aka SCSI Remote Protocol) fabric module (ib_srpt) for Mellanox HCAs is supported.

The InfiniBand/SRP fabric module went upstream into the Linux 3.3 kernel on 1/18/2012.[1]

Contents

Overview

SRP is based on the The SCSI RDMA Protocol. RDMA defines a SCSI mapping onto the Infiniband architecture and/or functionally similar cluster protocols, and generally allows higher throughput and lower latency than TCP/IP based communication. RDMA is only possible with network adapters that have hardware support for RDMA. Examples of such network adapters are InfiniBand HCAs and 10 GbE network adapters with iWARP support. While the SRP protocol has been designed to use RDMA networks efficiently, it is also possible to implement the SRP protocol over networks that do not support RDMA.

SRP is easier to implement than iSER, but iSER offers more management functionality, e.g. the target discovery infrastructure enabled by the iSCSI protocol.

Configuration

Use targetcli from Datera, Inc. to configure SRP targets.

Setup

Getting the sources

= Initial build: git setup

git clone lio-core from git://git.kernel.org/pub/scm/linux/kernel/git/nab/lio-core-2.6.git as follows:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/nab/lio-core-2.6.git
cd lio-core-2.6
git checkout --track -b tcm_ib_srpt-38 origin/tcm_ib_srpt-38
git pull origin tcm_ib_srpt-38
cp <your_working_config_file> .config
make menuconfig

In menuconfig:

<M> Generic Target Core Mod (TCM) and ConfigFS Infrastructure  --->
            --- Generic Target Core Mod (TCM) and ConfigFS Infrastructure     
                   <M>   TCM/IBLOCK Subsystem Plugin for Linux/BLOCK              
                   <M>   TCM/FILEIO Subsystem Plugin for Linux/VFS           
                   <M>   TCM/pSCSI Subsystem Plugin for Linux/SCSI                                        
                   < >   TCM/STGT Subsystem Plugin                             
                   <M>   Linux-iSCSI.org iSCSI Target Mode Stack
                   [ ]   LIO-Target iscsi_debug.h ring buffer messages
                   [ ]   LIO-Target ErrorRecoveryLevel debug code                   
<M> InfiniBand support  --->       
                 --- InfiniBand support                                                                   
                 <M>   InfiniBand userspace MAD support                                                  
                 <M>   InfiniBand userspace access (verbs and CM)                                       
                 <M>   Mellanox HCA support                                                             
                 < >   QLogic PCIe HCA support                                                          
                 < >   Ammasso 1100 HCA support                                                          
                 <M>   Mellanox ConnectX HCA support                                                    
                 < >   NetEffect RNIC Driver                                                          
                 <M>   IP-over-InfiniBand                                                                
                 [ ]     IP-over-InfiniBand Connected Mode support                                      
                 -*-     IP-over-InfiniBand debugging                                                    
                 [ ]       IP-over-InfiniBand data path debugging                                        
                 < >   InfiniBand SCSI RDMA Protocol                                                    
                 <M>   InfiniBand SCSI RDMA Protocol target support                                      
                 < >   iSCSI Extensions for RDMA (iSER)            

If CONFIGFS_FS=y is set, a manual mount is required after the kernel boots:

mount -t configfs configfs /sys/kernel/config

Subsequent builds: git pull

Go to the directory where your lio-core kernel is located, and:

cd <lio-core-2.6>
git pull origin tcm_ib_srpt-38
make clean
make oldconfig

Building

Kernel and modules:

make
make modules
make modules_install
make install

Installing

New Linux kernel

Use lio-core-2.6 kernel:

Then reboot:

reboot

and verify that the live kernel is indeed your new lio-core-2.6.git/tcm_ib_srpt-38 kernel:

uname -a 

Loading LIO Target

Verify the presence of the following required modules with modprobe: ib_srpt, target_core_mod, configfs, e.g.:

modprobe ib_srpt srp_max_req_size=4200

Normally all these modules should be loaded. If one is missing, you can load it as follows:

modprobe configfs
# Or do the mount command as shown above (if .config file has CONFIGFS_FS=y)
modprobe target_core_mod
modprobe ib_srpt srp_max_req_size=4200

Starting LIO Target

The target will be automatically started if targetcli-frozen is installed. Otherwise, start the target manually:

/etc/init.d/target start

To stop target:

/etc/init.d/target stop

To check the target status:

/etc/init.d/target status

Results

configfs

Please see SCSI RDMA Protocol/configFS for the running layout of a /sys/kernel/config/target/srpt configuration.

lsmod output

Module                  Size  Used by
ib_srpt                26028  6 
ib_umad                 9422  8 
iscsi_target_mod      201391  1 
mlx4_ib                38241  0 
target_core_pscsi      11045  0 
target_core_file        6307  1 
target_core_iblock      6606  1 
target_core_mod       206442  28 ib_srpt,iscsi_target_mod,target_core_pscsi,target_core_file,target_core_iblock
configfs               19153  2 target_core_mod
ib_iser                23172  0 
rdma_cm                23833  1 ib_iser
ib_cm                  24878  2 ib_srpt,rdma_cm
iw_cm                   6019  1 rdma_cm
ib_sa                  15537  2 rdma_cm,ib_cm
ib_mad                 30330  5 ib_srpt,ib_umad,mlx4_ib,ib_cm,ib_sa
ib_core                39716  9 ib_srpt,ib_umad,mlx4_ib,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad
ib_addr                 3982  1 rdma_cm
ipv6                  242427  21 ib_addr
iscsi_tcp               7729  0 
libiscsi_tcp           10485  1 iscsi_tcp
libiscsi               29466  3 ib_iser,iscsi_tcp,libiscsi_tcp
scsi_transport_iscsi    24624  4 ib_iser,iscsi_tcp,libiscsi
dm_multipath           14016  0 
scsi_dh                 4656  1 dm_multipath
</code>

==== Interrupt stats ====

MSI-X interrupt stats for target side operation:

<pre>
lenny64guest0:/usr/src/lio-core-2.6.git# cat /proc/interrupts | grep mlx4
43:   16864937   16864950   16864795   16864897   PCI-MSI-edge      mlx4-comp-0@pci:0000:00:05.0
44:          0          0          0          0   PCI-MSI-edge      mlx4-comp-1@pci:0000:00:05.0
45:          0          0          0          0   PCI-MSI-edge      mlx4-comp-2@pci:0000:00:05.0
46:          0          0          0          0   PCI-MSI-edge      mlx4-comp-3@pci:0000:00:05.0
47:       1377       1308       1338       1322   PCI-MSI-edge      mlx4-async@pci:0000:00:05.0

ibsrpdm output

# ibsrpdm 
IO Unit Info:
    port LID:        0002
    port GID:        fe800000000000000002c903000e8acd
    change ID:       0001
    max controllers: 0x10
    controller[  1]
        GUID:      0002c903000e8acc
        vendor ID: 000002
        device ID: 00673c
        IO class : 0100
        ID:        Linux SRP target
        service entries: 1
        service[  0]: 0002c903000e8acc / SRP.T10:0002c903000e8acc

Initiator

Kernel ring buffer

Initiator side ring buffer output during initiator login:

scsi7 : SRP.T10:0002C903000E8ACC
scsi 7:0:0:0: Direct-Access     LIO-ORG  RAMDISK-MCP      4.0  PQ: 0 ANSI: 5
sd 7:0:0:0: Attached scsi generic sg2 type 0
sd 7:0:0:0: [sdb] 524288 512-byte logical blocks: (268 MB/256 MiB)
sd 7:0:0:0: [sdb] Write Protect is off
sd 7:0:0:0: [sdb] Mode Sense: 2f 00 00 00
sd 7:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
 sdb: unknown partition table
sd 7:0:0:0: [sdb] Attached SCSI disk

Device identifiers

sg_inq -i initiator side output for EVPD=0x83 device identifiers:

# sg_inq -i /dev/sdb
VPD INQUIRY: Device Identification page
  Designation descriptor number 1, descriptor length: 20
    designator_type: NAA,  code_set: Binary
    associated with the addressed logical unit
      NAA 6, IEEE Company_id: 0x1405
      Vendor Specific Identifier: 0x929cc061d
      Vendor Specific Identifier Extension: 0xe082d4d98d810cdc
      [0x6001405929cc061de082d4d98d810cdc]
  Designation descriptor number 2, descriptor length: 61
    designator_type: T10 vendor identification,  code_set: ASCII
    associated with the addressed logical unit
      vendor id: LIO-ORG
      vendor specific: RAMDISK-MCP:929cc061-e082-4d98-810c-c90b421bd066
  Designation descriptor number 3, descriptor length: 8
    transport: Remote Direct Memory Access (RDMA)
    designator_type: Relative target port,  code_set: Binary
    associated with the target port
      Relative target port: 0x2
  Designation descriptor number 4, descriptor length: 8
    transport: Remote Direct Memory Access (RDMA)
    designator_type: Target port group,  code_set: Binary
    associated with the target port
      Target port group: 0x0
  Designation descriptor number 5, descriptor length: 8
    designator_type: Logical unit group,  code_set: Binary
    associated with the addressed logical unit
      Logical unit group: 0x0
  Designation descriptor number 6, descriptor length: 32
    transport: Remote Direct Memory Access (RDMA)
    designator_type: SCSI name string,  code_set: UTF-8
    associated with the target port
      SCSI name string:
      0x2c903000e8acc,t,0x0001

Specifications

SRP was not approved as an official standard. The following specifications are available as available as T10 Working Drafts:

Timeline

Timeline of the LinuxIO
Release Details 2011 2012 2013 2014 2015
123456789101112 123456789101112 123456789101112 123456789101112 123456789101112
4.x Version 4.0 4.1
Feature LIO Core Loop back FCoE iSCSI Perf SRP
CM WQ FC
USB
1394
vHost Perf Misc 16 GFC iSER Misc VAAI Misc DIF Core
NPIV
DIF iSER DIF FC vhost TCMU Xen Misc Misc virtio 1.0 Misc NVMe OF
Linux 2.6.38 2.6.39 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18 3.19 3.20 3.21 3.22

See also

Notes

  1. Linus Torvalds (1/18/2012). "InfiniBand/SRP merge". lkml.org. 

External links

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox
Google AdSense