InfiniBand

The Linux SCSI Target Wiki

Revision as of 00:06, 16 September 2012 by Marcf (Talk | contribs)
Jump to: navigation, search
LIO Target
Logo
Mellanox Technologies, Ltd.
Mellanox Infiniband SRP fabric module
Original author(s) Vu Pham
Bart Van Assche
Nicholas Bellinger
Developer(s) Mellanox Technologies, Ltd.
Initial release March 18, 2012 (2012-03-18)
Stable release 4.1.0 / June 20, 2012;
7 years ago
 (2012-06-20)
Preview release 4.2.0-rc5 / June 28, 2012;
7 years ago
 (2012-06-28)
Development status Production
Written in C
Operating system Linux
Type Fabric module
License GNU General Public License
Website mellanox.com
See Target for a complete overview over all fabric modules.

InfiniBand provides the target for various IB Host Channel Adapters (HCAs).

Contents

Overview

InfiniBand is an industry standard, channel-based, switched-fabric, interconnect architecture for servers. It is used predominantly in high-performance computing (HPC), and recently has enjoyed increasing popularity for SANs. Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable.

The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. InfiniBand forms a superset of the Virtual Interface Architecture.

The InfiniBand fabric module for the Unified Target supports the SCSI RDMA Protocol (SRP).

Hardware support

The following Mellanox InfiniBand HCAs are supported:

The InfiniBand/SRP fabric module (srpt.ko, Linux kernel driver database) for the Unified Target was released with Linux kernel 3.3 on March 18, 2012 (2012-03-18).[1]

targetcli

targetcli from Datera, Inc. is used to configure InfiniBand targets. targetcli aggregates service modules via a core library, and exports them through an API to the Unified Target, to provide a unified single-node SAN configuration shell, independently of the underlying fabric(s).

Startup

targetcli is invoked by running targetcli as root from the command prompt of the underlying OS shell.

# targetcli
Welcome to targetcli:

 Copyright (c) 2012 by RisingTide Systems LLC.
 All rights reserved.

Visit us at http://www.risingtidesystems.com.

Using ib_srpt fabric module.
Using qla2xxx fabric module.
Using iscsi fabric module.
Using loopback fabric module.

/> ib_srpt/ info
Fabric module name: ib_srpt
ConfigFS path: /sys/kernel/config/target/srpt
Allowed WWN list type: free
Fabric module specfile: /var/target/fabric/ib_srpt.spec
Fabric module features: acls
Corresponding kernel module: ib_srpt
/>

Upon targetcli initialization, the underlying RTSlib loads the installed fabric modules, and creates the corresponding ConfigFS mount points (at /sys/kernel/config/target/<fabric>), as specified by the associated spec files (located in /var/target/fabric/fabric.spec).

Display the object tree

Use ls to list the object hierarchy, which is initially empty:

/> ls
o- / ..................................................................... [...]
  o- backstores .......................................................... [...]
  | o- fileio ............................................... [0 Storage Object]
  | o- iblock ............................................... [0 Storage Object]
  | o- pscsi ................................................ [0 Storage Object]
  | o- rd_dr ................................................ [0 Storage Object]
  | o- rd_mcp ............................................... [0 Storage Object]
  o- ib_srpt ........................................................ [0 Target]
  o- iscsi .......................................................... [0 Target]
  o- loopback ....................................................... [0 Target]
  o- qla2xxx ........................................................ [0 Target]
/>

Template:Message/tip

Per default, auto_cd_after_create is set to true, which automatically enters the object context (or working directory) of new objects after their creation. Set auto_cd_after_create=false to prevent RTSadmin from automatically entering the object context to new objects after their creation:

/> set global auto_cd_after_create=false
Parameter auto_cd_after_create is now 'false'.
/>

Create a backstore

Enter the top-level backstore object, and create one (storage object) using IBLOCK or FILEIO type devices.

For instance, create an IBLOCK backstore from a /dev/sdb block device. Note that this device can be any TYPE_DISK block-device, and it can also use /dev/disk/by-id/ symlinks:

/> cd backstores/
/backstores> iblock/ create name=my_disk dev=/dev/sdb
Generating a wwn serial.
Created iblock storage object my_disk using /dev/sdb.
Entering new node /backstores/iblock/my_disk.
/backstores/iblock/my_disk>

RTSadmin automatically creates a WWN serial ID for the backstore device and then changes the working context to it.

The resulting object hierarchy looks as follows (displayed from the root object):

/> ls
o- / ..................................................................... [...]
  o- backstores .......................................................... [...]
  | o- fileio ............................................... [0 Storage Object]
  | o- iblock ............................................... [1 Storage Object]
  | | o- my_disk .......................................... [/dev/sdb activated]
  | o- pscsi ................................................ [0 Storage Object]
  | o- rd_dr ................................................ [0 Storage Object]
  | o- rd_mcp ............................................... [0 Storage Object]
  o- ib_srpt ........................................................ [0 Target]
  o- iscsi .......................................................... [0 Target]
  o- loopback ....................................................... [0 Target]
  o- qla2xxx ........................................................ [0 Target]
/>

Instantiate a target

The InfiniBand ports that are available on the storage array are presented in the WWN context with the following WWPNs, for instance:

Instantiate an InfiniBand target, in this example for SRP over Mellanox Connect-X HCAs, on the existing IBLOCK backstore device my_disk (as set up in targetcli):

/backstores/iblock/my_disk> /ip_srpt create 0x00000000000000000002c903000e8acd
Created target 0x00000000000000000002c903000e8acd.
Entering new node /ib_srpt/0x00000000000000000002c903000e8acd.
/ib_srpt/0x00...2c903000e8acd>

targetcli automatically changes the working context to the resulting tagged Endpoint.

Export LUNs

Declare a LUN for the backstore device, to form a valid SAN storage object:

/ib_srpt/0x00...2c903000e8acd> luns/ create /backstores/iblock/my_disk
Selected LUN 0.
Successfully created LUN 0.
Entering new node /ib_srpt/0x00000000000000000002c903000e8acd/luns/lun0.
/ib_srpt/0x00...acd/luns/lun0>

targetcli per default automatically assigns the default ID '0' to the LUN, and then changes the working context to the SAN storage object. The target is now created, and exports /dev/sdb as LUN 0.

Return to the underlying Endpoint as the working context, as no attributes need to be set or modified for standard LUNs:

/ib_srpt/0x00...act/luns/lun0> cd <
Taking you back to /ib_srpt/0x00000000000000000002c903000e8acd.
/ib_srpt/0x00...2c903000e8acd>

Define access rights

Configure the access rights to allow logins from initiators. This requires setting up individual access rights for each initiators, based on its WWPN.

Determine the WWPN for the respective InfiniBand initiator. For Linux initiator systems, e.g., use:

# cat /sys/class/infiniband/*/ports/*/gids/0 | sed -e s/fe80/0x0000/ -e 's/\://g'

For a simple setup, allow access to the initiator with the WWPN as determined above:

/ib_srpt/0x00...2c903000e8acd> acls/ create 0x00000000000000000002c903000e8be9
Successfully created Node ACL for 0x00000000000000000002c903000e8be9.
Created mapped LUN 0.
Entering new node /ib_srpt/0x00000000000000000002c903000e8acd/acls/
0x00000000000000000002c903000e8be9.
/ib_srpt/0x00...2c903000e8be9> cd /

The targetcli shell then automatically adds the appropriate mapped LUNs per default.

Display the object tree

The resulting InfiniBand SAN object hierarchy looks as follows (displayed from the root object):

/> ls
o- / ..................................................................... [...]
  o- backstores .......................................................... [...]
  | o- fileio ............................................... [0 Storage Object]
  | o- iblock ............................................... [1 Storage Object]
  | | o- my_disk .......................................... [/dev/sdb activated]
  | o- pscsi ................................................ [0 Storage Object]
  | o- rd_dr ................................................ [0 Storage Object]
  | o- rd_mcp ............................................... [0 Storage Object]
  o- ib_srpt ........................................................ [1 Target]
  | o- 0x00000000000000000002c903000e8acd ............................ [enabled]
  |   o- acls .......................................................... [1 ACL]
  |   | o- 0x00000000000000000002c903000e8be9 ................... [1 Mapped LUN]
  |   |   o- mapped_lun0 ........................................... [lun0 (rw)]
  |   o- luns .......................................................... [1 LUN]
  |     o- lun0 .................................... [iblock/my_disk (/dev/sdb)]
  o- iscsi .......................................................... [0 Target]
  o- loopback ....................................................... [0 Target]
  o- qla2xxx ........................................................ [0 Target]
/>

Persist the configuration

The target configuration can be persisted across OS reboots by using saveconfig from the root context:

/> saveconfig
WARNING: Saving rtsnode1 current configuration to disk will overwrite your boot settings.
The current target configuration will become the default boot config.
Are you sure? Type 'yes': yes
Making backup of srpt/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
Successfully updated default config /etc/target/srpt_start.sh
Making backup of qla2xxx/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
Successfully updated default config /etc/target/qla2xxx_start.sh
Making backup of loopback/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
Successfully updated default config /etc/target/loopback_start.sh
Making backup of LIO-Target/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
Successfully updated default config /etc/target/lio_backup-2012-02-27_23:19:37.660264.sh
Making backup of Target_Core_Mod/ConfigFS with timestamp: 2012-02-27_23:19:37.660264
Successfully updated default config /etc/target/tcm_backup-2012-02-27_23:19:37.660264.sh
Generated Target_Core_Mod config: /etc/target/backup/tcm_backup-2012-02-27_23:19:37.660264.sh
Successfully updated default config /etc/target/lio_start.sh
Successfully updated default config /etc/target/tcm_start.sh
/>

Template:Message/warning

Spec file

RTS spec files define the fabric-dependent feature set, capabilities and available target ports of the specific underlying fabric.

In particular, the InfiniBand spec file /var/target/fabric/ib_srpt.spec is included via RTSlib. WWN values are extracted via /sys/class/infiniband/*/ports/*/gids/0 in wwn_from_files_filter below, and are presented in the targetcli WWN context to register individual InfiniBand port GUIDs.

# WARNING: This is a draft specfile supplied for demo purposes only.

# The ib_srpt fabric module uses the default feature set.
features = acls

# The module uses hardware addresses from there
wwn_from_files = /sys/class/infiniband/*/ports/*/gids/0

# Transform 'fe80:0000:0000:0000:0002:1903:000e:8acd' WWN notation to
# '0x00000000000000000002c903000e8acd'
wwn_from_files_filter = "sed -e s/fe80/0x0000/ -e 's/\://g'"

# Non-standard module naming scheme
kernel_module = ib_srpt

# The configfs group is standard
configfs_group = srpt

Protocols

A brief overview over relevant or related InfiniBand protocols:

Specifications

SRP was not approved as an official standard. The following specifications are available as available as T10 Working Drafts:

Glossary

RFCs

Timeline

Timeline of the LinuxIO
Release Details 2011 2012 2013 2014 2015
123456789101112 123456789101112 123456789101112 123456789101112 123456789101112
4.x Version 4.0 4.1
Feature LIO Core Loop back FCoE iSCSI Perf SRP
CM WQ FC
USB
1394
vHost Perf Misc 16 GFC iSER Misc VAAI Misc DIF Core
NPIV
DIF iSER DIF FC vhost TCMU Xen Misc Misc virtio 1.0 Misc NVMe OF
Linux 2.6.38 2.6.39 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18 3.19 3.20 3.21 3.22

See also

Notes

  1. Linus Torvalds (2012-03-18). "Linux 3.3". lkml.org. 
  2. Tom Talpey, et al. (8/26/2009). "Remote Direct Memory Access over the Converged Enhanced Ethernet Fabric: Evaluating the Options". IEEE Hot Interconnects 17. 

Wikipedia entries

External links

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox
Google AdSense