FreeNAS users don't run their cards with the
"option ROM" and it may cause unknown issues to have it installed. Less,
it seems, is more.
sas2flash.efi -o -f 2118it.bin -b mptsas2.rom
For my second M1015 card I used this SAS address:
sas2flash.efi -o -f 2118it.bin
sas2flash.efi -o -sasadd 500605B002B086C0
For my third M1015 card I used this SAS address:
sas2flash.efi -o -f 2118it.bin
sas2flash.efi -o -sasadd 500605B00260A130
For my fourth M1015 card I used this SAS address:
sas2flash.efi -o -f 2118it.bin
sas2flash.efi -o -sasadd 500605B002626E00
For my fifth M1015 card I used this SAS address:
sas2flash.efi -o -f 2118it.bin
sas2flash.efi -o -sasadd 500605B002AFA300
For my sixth M1015 card I used this SAS address:
sas2flash.efi -o -f 2118it.bin
sas2flash.efi -o -sasadd 500605B002611DC0
If you were using a slot on the the Asus P8Z77-V motherboard to perform
the flashing, and you had another disk controller card in the same slot
on that motherboard before you began, then when you put that card back,
the original boot order that the motherboard used will likley now be
"forgotten" because the card had temporarily been removed (even though
you didn't change the boot order, the motherboard noticed that those
drives were no longer present while the M1015 card was installed). So to
set the boot order back to how it was before, you can go into the Asus
P8Z77-V UEFI by pressing Del
and in the UEFI screen clicking the Exit/Advanced Mode
button and clicking on the Boot Menu
button where you'll find a list entry with the name Hard Drive BBS Priorities
,
click on it. In this page, by clicking on any of the "drive slots"
shown, you can choose the order the drives boot in. For me, it needs to
be Boot Option #1
as WDC WD 1600JD-75HBBO
and Boot Option #2
as A050 Samsung SSD 840 EVO 500GB
, this is because grub
is installed on the Western Digital drive (for dual-boot with Ubuntu), and Microsoft Windows 7
is installed on the Samsung SSD.
- IBM M1015 (front):
- IBM M1015 (back, shows the green label with the SAS address to use):
Linux Driver for the LSI 9211-8i card is mpt2sas
My distro is as follows:
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.10
DISTRIB_CODENAME=utopic
DISTRIB_DESCRIPTION="Ubuntu 14.10"
The boot-time messages are as follows:
[ 0.905756] mpt2sas version 16.100.00.00 loaded
[ 0.905933] scsi0 : Fusion MPT SAS Host
[ 0.906217] mpt2sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (16423736 kB)
[ 0.906321] mpt2sas 0000:05:00.0: irq 48 for MSI/MSI-X
[ 0.906341] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 48
[ 0.906399] mpt2sas0: iomem(0x00000000f0cc0000), mapped(0xffffc90005b90000), size(16384)
[ 0.906470] mpt2sas0: ioport(0x000000000000e000), size(256)
Manually doing a
modprobe -r mpt2sas
modprobe mpt2sas
causes these messages if you do NOT have any SFF-8087 connectors plugged in (i.e. with no drives connected to the card):
[ 1372.098618] mpt2sas version 16.100.00.00 loaded
[ 1372.098928] scsi10 : Fusion MPT SAS Host
[ 1372.099528] mpt2sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (16423736 kB)
[ 1372.099609] mpt2sas 0000:05:00.0: irq 48 for MSI/MSI-X
[ 1372.099654] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 48
[ 1372.099660] mpt2sas0: iomem(0x00000000f0cc0000), mapped(0xffffc90005b90000), size(16384)
[ 1372.099664] mpt2sas0: ioport(0x000000000000e000), size(256)
[ 1372.553651] mpt2sas0: Allocated physical memory: size(7445 kB)
[ 1372.553656] mpt2sas0: Current Controller Queue Depth(3307), Max Controller Queue Depth(3432)
[ 1372.553657] mpt2sas0: Scatter Gather Elements per IO(128)
[ 1372.787386] mpt2sas0: LSISAS2008: FWVersion(16.00.00.00), ChipRevision(0x03), BiosVersion(00.00.00.00)
[ 1372.787393] mpt2sas0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[ 1372.787491] mpt2sas0: sending port enable !!
[ 1372.791393] mpt2sas0: host_add: handle(0x0001), sas_addr(0x500605b002b086c0), phys(8)
[ 1372.806556] mpt2sas0: port enable: SUCCESS
For me, initially the drives were not being seen (because I hadn't
pushed the SFF-8087 connector hard enough into its socket on the card,
as I was worried about damaging the fragile socket or the flimsy-looking
connector). Anyways, now the drives are showing up, and
hardware burn-in and validation testing with ZFS can begin in earnest on
Linux (in preparation for running the card in a FreeNAS? server).
It's hard to imagine how this kind of SFF-8087 socket design was chosen for use in enterprise servers.
mpt2sas version 16.100.00.00 loaded
scsi13 : Fusion MPT SAS Host
mpt2sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (16423736 kB)
mpt2sas 0000:05:00.0: irq 48 for MSI/MSI-X
mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 48
mpt2sas0: iomem(0x00000000f0cc0000), mapped(0xffffc90005b90000), size(16384)
mpt2sas0: ioport(0x000000000000e000), size(256)
mpt2sas0: Allocated physical memory: size(7445 kB)
mpt2sas0: Current Controller Queue Depth(3307), Max Controller Queue Depth(3432)
mpt2sas0: Scatter Gather Elements per IO(128)
mpt2sas0: LSISAS2008: FWVersion(16.00.00.00), ChipRevision(0x03), BiosVersion(00.00.00.00)
mpt2sas0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
mpt2sas0: sending port enable !!
mpt2sas0: host_add: handle(0x0001), sas_addr(0x500605b002b086c0), phys(8)
scsi 13:0:0:0: Direct-Access ATA ST32000542AS CC34 PQ: 0 ANSI: 6
scsi 13:0:0:0: SATA: handle(0x0009), sas_addr(0x4433221107000000), phy(7), device_name(0x5000c50032bef27a)
scsi 13:0:0:0: SATA: enclosure_logical_id(0x500605b002b086c0), slot(4)
scsi 13:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
scsi 13:0:0:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
mpt2sas0: port enable: SUCCESS
sd 13:0:0:0: Attached scsi generic sg11 type 0
sd 13:0:0:0: [sdj] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
sd 13:0:0:0: [sdj] Write Protect is off
sd 13:0:0:0: [sdj] Mode Sense: 7f 00 10 08
sd 13:0:0:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA
sdj: sdj1 sdj2 sdj3
sd 13:0:0:0: [sdj] Attached SCSI disk
An initial sequential read test (just using dd
on two drives to copy a raw partition on each of them to /dev/null
) on two Western Digital 6TB WD60EFRX red drives shows iostat
like this, which seems OK, averaging 178MB/s (which is the max these drives are capable of for sequential read).
avg-cpu: %user %nice %system %iowait %steal %idle
2.33 0.00 17.36 32.90 0.00 47.41
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 17.00 0.00 132.00 0 132
sdb 1415.00 181120.00 0.00 181120 0
sdc 1471.00 188288.00 0.00 188288 0
avg-cpu: %user %nice %system %iowait %steal %idle
2.79 0.00 16.50 39.34 0.00 41.37
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 14.00 12.00 104.00 12 104
sdb 1282.00 164096.00 0.00 164096 0
sdc 1375.00 176000.00 0.00 176000 0
avg-cpu: %user %nice %system %iowait %steal %idle
3.29 0.00 16.46 30.38 0.00 49.87
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 2.00 0.00 12.00 0 12
sdb 1455.00 186240.00 0.00 186240 0
sdc 1494.00 191232.00 0.00 191232 0
Another sequential read/write test (just using dd
to copy
the contents of a raw partition into an ext4 filesystem on another
drive) shows the transfer averaging around 161 MB/s which seems fine.
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 8.84 37.88 0.00 53.28
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 15.00 0.00 76.00 0 76
sdb 1361.00 174208.00 0.00 174208 0
sdc 356.00 0.00 182272.00 0 182272
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 9.16 30.79 0.00 60.05
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 3.00 8.00 4.00 8 4
sdb 1421.00 181888.00 0.00 181888 0
sdc 357.00 0.00 182784.00 0 182784
avg-cpu: %user %nice %system %iowait %steal %idle
0.25 0.00 8.59 30.81 0.00 60.35
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 0.00 0.00 0.00 0 0
sdb 1318.00 168704.00 0.00 168704 0
sdc 355.00 0.00 181760.00 0 181760
The FLReset+
flag.
For PCI Passthrough (known as IOMMU in Xen), the FLReset+
flag needs to be seen in the lspci
output, which it is for these cards.
First IBM M1015 card:
lspci -vvnn
03:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
Subsystem: LSI Logic / Symbios Logic Device [1000:3020]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Interrupt: pin A routed to IRQ 16
Region 0: I/O ports at c000 [size=256]
Region 1: Memory at c10c0000 (64-bit, non-prefetchable) [size=16K]
Region 3: Memory at c0c80000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at c0c00000 [disabled] [size=512K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [d0] Vital Product Data
Unknown small resource type 00, will not decode more.
Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
Vector table: BAR=1 offset=00002000
PBA: BAR=1 offset=00003800
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [138 v1] Power Budgeting <?>
Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: 0072
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000c10c4000 (64-bit, non-prefetchable)
Region 2: Memory at 00000000c0cc0000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Kernel driver in use: mpt2sas
Second IBM M1015 card:
lspci -vvnn
09:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 02)
Subsystem: LSI Logic / Symbios Logic Device [1000:3020]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Interrupt: pin A routed to IRQ 31
Region 0: I/O ports at d000 [size=256]
Region 1: Memory at c0ac0000 (64-bit, non-prefetchable) [size=16K]
Region 3: Memory at c0680000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at c0600000 [disabled] [size=512K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [d0] Vital Product Data
Unknown small resource type 00, will not decode more.
Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
Vector table: BAR=1 offset=00002000
PBA: BAR=1 offset=00003800
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [138 v1] Power Budgeting <?>
Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
IOVSta: Migration-
Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: 0072
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000c0ac4000 (64-bit, non-prefetchable)
Region 2: Memory at 00000000c06c0000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Kernel driver in use: mpt2sas
Further info
See here for further info about the M1015:
- The M1015 native firmware "personality" is the same as an LSI
9240-8i, with the exception that Raid5 functionality is disabled unless
you install a "key" onto the card.
- The M1015 can be flashed successfully with the LSI 9240-8i
firware from the LSI website using megacli, although access to Raid5 is
still controlled by the raid key. In fact, there is some evidence that
IBMs native load and LSIs 9240 load are identical, though that is
difficult to confirm because IBMs firmware always seems to have a minor
rev level difference which makes a simple bitwise compare impossible.
- The M1015 can also be flashed with 9210-8i or 9211-8i using a procedure detailed earlier in this thread. It can apparently accept either the IR or IT load for these cards.
- There is no "IT Mode" for the native M1015 or 9240 drivers.
It simply is not needed. The reason people wanted the IT load when
running as a 9211 is because the 9211 doesn't really do single disk
passthrough correctly - it builds the disk as a single-drive raid 0,
which puts nasty little headers on the disk and mucks up your ability to
share disks directly with other systems. The 9240 firmware does not do
this - any drive not part of a raid is just passed through directly as a
single drive - and no special 'IT mode' driver is necessary.
- The only reason to flash this card to a 9211 personality is
to use it with an OS that does not support the 9240 (e.g., freeBSD). For
others (Windows, Linux, Solaris, etc) the 9240 drivers are available
and work well - some would suggest work better (e.g., the Solaris imr
drivers for the 9240 do a better job of passing through disk temps and
SMART).
- john4200 has been using the 9240-8i firmware on his M1015 with the
megaraid_sas
linux driver, and found the megaraid_sas
linux driver to be less buggy than the mpt2sas
driver. The mpt2sas
driver is the one that's used with the IT firmware. Some bug reports exist for the mpt2sas
driver and IT firmware LSI cards, they're driver-related rather than firmware-related.
- When reading about the M1015 card, if you're wondering what
megacli
is, that's only for if you want to update the 9240-8i kind of firmware corresponding to this card's "normal" megaraid_sas
Linux driver (since the M1015 is a megaraid card), rather than
cross-flash it to turn it into an LSI Logic 9211.8i card (also known as
an "LSI SAS2008" card) using the mpt2sas
driver. If you want to change the M1015 firmware so that you can use the mpt2sas
Linux driver (like an LSI Logic 9211.8i), then you will need megarec
to erase the old firmware and sas2flash
to install the new firmware (this is the "cross-flashing" step), as described on this page. You should not try to use sas2flash
on a card that is still a megaraid card, obviously.
Getting into an EFI shell
- Getting into "Advanced Mode" in the P8Z77-V EFI:
- Need to change "Secure Boot" to allow us into UEFI shell...:
- Change to "Other OS":
- Change to "Other OS":
- In the "Exit" menu, see the "Lauch EFI Shell from filesystem device" menu item...:
- See here for a video of launching the EFI shell...
- See here for a video of clearing the existing firmware using the
megarec -cleanflash 0
command.
- See here for a video of how the
magarec
command sometimes fails.
- Output during flashing the LSI P16 firmware onto the M1015 using the "sas2flash.efi -o -f 2118it.bin" command.:
- More output during flashing the LSI P16 firmware onto the M1015 using the "sas2flash.efi -o -f 2118it.bin" command.:
- Setting the SAS address of the M1015 card using the command "sas2flash.efi -o -sasadd 500605B0026A6750":
- Checking the firmware version of the card using the "sas2flash.efi
-listall" command, yes it's the P16 firmware, listed as 16.00.00.00: