SCSI RDMA Protocol

The Linux SCSI Target Wiki

Revision as of 02:51, 27 March 2011 by Admin (Talk | contribs)
Jump to: navigation, search
LIO Target
Logo
Mellanox Technologies, Ltd.
Mellanox Infiniband SRP fabric module
Original author(s) Vu Pham
Developer(s) Mellanox Technologies, Ltd.
Initial release March 18, 2012 (2012-03-18)
Stable release 4.1.0 / June 20, 2012;
7 years ago
 (2012-06-20)
Preview release 4.2.0-rc5 / June 28, 2012;
7 years ago
 (2012-06-28)
Development status Beta
Written in C
Operating system Linux
Type Fabric module
License GNU General Public License
Website www.mellanox.com
See Target for a complete overview over all fabric modules.

An SCSI RDMA Protocol (SRP, aka SCSI Remote Protocol) fabric module (tcm_ib_srpt) for Mellanox HCAs is available upon request. SRP defines a SCSI protocol mapping onto the Infiniband architecture and/or functionally similar cluster protocols.

The use of RDMA generally allows higher throughput and lower latency than TCP/IP based communication. RDMA is only possible with network adapters that support RDMA in hardware. Examples of such network adapters are InfiniBand HCAs and 10 GbE network adapters with iWARP support. While the SRP protocol has been designed to use RDMA networks efficiently, it is also possible to implement the SRP protocol over networks that do not support RDMA.

SRP is easier to implement than iSER, but iSER offers more management functionality, e.g. the target discovery infrastructure enabled by the iSCSI protocol.

Contents

Status

The SRP driver is currently being tested and optimized.

Specifications

SRP was not approved as an official standard. The following specifications are available as available as T10 Working Drafts:

Results

configfs

Please see SCSI RDMA Protocol/configFS for the running layout of a /sys/kernel/config/target/srpt configuration

lsmod output

 Module                  Size  Used by
 ib_srpt                26028  6 
 ib_umad                 9422  8 
 iscsi_target_mod      201391  1 
 mlx4_ib                38241  0 
 target_core_pscsi      11045  0 
 target_core_file        6307  1 
 target_core_iblock      6606  1 
 target_core_mod       206442  28 ib_srpt,iscsi_target_mod,target_core_pscsi,target_core_file,target_core_iblock
 configfs               19153  2 target_core_mod
 ib_iser                23172  0 
 rdma_cm                23833  1 ib_iser
 ib_cm                  24878  2 ib_srpt,rdma_cm
 iw_cm                   6019  1 rdma_cm
 ib_sa                  15537  2 rdma_cm,ib_cm
 ib_mad                 30330  5 ib_srpt,ib_umad,mlx4_ib,ib_cm,ib_sa
 ib_core                39716  9 ib_srpt,ib_umad,mlx4_ib,ib_iser,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad
 ib_addr                 3982  1 rdma_cm
 ipv6                  242427  21 ib_addr
 iscsi_tcp               7729  0 
 libiscsi_tcp           10485  1 iscsi_tcp
 libiscsi               29466  3 ib_iser,iscsi_tcp,libiscsi_tcp
 scsi_transport_iscsi    24624  4 ib_iser,iscsi_tcp,libiscsi
 dm_multipath           14016  0 
 scsi_dh                 4656  1 dm_multipath

Interrupt stats

MSI-X interrupt stats for target side operation:

 lenny64guest0:/usr/src/lio-core-2.6.git# cat /proc/interrupts | grep mlx4
 43:   16864937   16864950   16864795   16864897   PCI-MSI-edge      mlx4-comp-0@pci:0000:00:05.0
 44:          0          0          0          0   PCI-MSI-edge      mlx4-comp-1@pci:0000:00:05.0
 45:          0          0          0          0   PCI-MSI-edge      mlx4-comp-2@pci:0000:00:05.0
 46:          0          0          0          0   PCI-MSI-edge      mlx4-comp-3@pci:0000:00:05.0
 47:       1377       1308       1338       1322   PCI-MSI-edge      mlx4-async@pci:0000:00:05.0

Initiator

Kernel ring buffer

Initiator side ring buffer output during initiator login:

 scsi7 : SRP.T10:0002C903000E8ACC
 scsi 7:0:0:0: Direct-Access     LIO-ORG  RAMDISK-MCP      4.0  PQ: 0 ANSI: 5
 sd 7:0:0:0: Attached scsi generic sg2 type 0
 sd 7:0:0:0: [sdb] 524288 512-byte logical blocks: (268 MB/256 MiB)
 sd 7:0:0:0: [sdb] Write Protect is off
 sd 7:0:0:0: [sdb] Mode Sense: 2f 00 00 00
 sd 7:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
  sdb: unknown partition table
 sd 7:0:0:0: [sdb] Attached SCSI disk

Device identifiers

sg_inq -i initiator side output for EVPD=0x83 device identifiers:

 truelife:/usr/src/lio-core-2.6.git# sg_inq -i /dev/sdb
 VPD INQUIRY: Device Identification page
   Designation descriptor number 1, descriptor length: 20
     designator_type: NAA,  code_set: Binary
     associated with the addressed logical unit
       NAA 6, IEEE Company_id: 0x1405
       Vendor Specific Identifier: 0x929cc061d
       Vendor Specific Identifier Extension: 0xe082d4d98d810cdc
       [0x6001405929cc061de082d4d98d810cdc]
   Designation descriptor number 2, descriptor length: 61
     designator_type: T10 vendor identification,  code_set: ASCII
     associated with the addressed logical unit
       vendor id: LIO-ORG
       vendor specific: RAMDISK-MCP:929cc061-e082-4d98-810c-c90b421bd066
   Designation descriptor number 3, descriptor length: 8
     transport: Remote Direct Memory Access (RDMA)
     designator_type: Relative target port,  code_set: Binary
     associated with the target port
       Relative target port: 0x2
   Designation descriptor number 4, descriptor length: 8
     transport: Remote Direct Memory Access (RDMA)
     designator_type: Target port group,  code_set: Binary
     associated with the target port
       Target port group: 0x0
   Designation descriptor number 5, descriptor length: 8
     designator_type: Logical unit group,  code_set: Binary
     associated with the addressed logical unit
       Logical unit group: 0x0
   Designation descriptor number 6, descriptor length: 32
     transport: Remote Direct Memory Access (RDMA)
     designator_type: SCSI name string,  code_set: UTF-8
     associated with the target port
       SCSI name string:
       0x2c903000e8acc,t,0x0001

ibsrpdm output

 truelife:/usr/src/lio-core-2.6.git# ibsrpdm 
 IO Unit Info:
     port LID:        0002
     port GID:        fe800000000000000002c903000e8acd
     change ID:       0001
     max controllers: 0x10
     controller[  1]
         GUID:      0002c903000e8acc
         vendor ID: 000002
         device ID: 00673c
         IO class : 0100
         ID:        Linux SRP target
         service entries: 1
             service[  0]: 0002c903000e8acc / SRP.T10:0002c903000e8acc

Timeline

Timeline of the LinuxIO
Release Details 2011 2012 2013 2014 2015
123456789101112 123456789101112 123456789101112 123456789101112 123456789101112
4.x Version 4.0 4.1
Feature LIO Core Loop back FCoE iSCSI Perf SRP
CM WQ FC
USB
1394
vHost Perf Misc 16 GFC iSER Misc VAAI Misc DIF Core
NPIV
DIF iSER DIF FC vhost TCMU Xen Misc Misc virtio 1.0 Misc NVMe OF
Linux 2.6.38 2.6.39 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18 3.19 3.20 3.21 3.22

See also

Notes

External links

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox
Google AdSense