Sunday, January 25, 2015

FreeBSD LSI SAS9211-8i HBA firmware notes

I'll be using this post to store information about LSI HBA firmware, with a focus on FreeBSD (but also drawing upon Linux information). It may also be useful for users of FreeNAS, PC-BSD, unRAID, Nexenta, or ZFSguru.

Why - SATA port density on a budget

If you are using ZFS, you do not need RAID -- you just need lots of fast SATA ports. To maximize the features of ZFS, it needs to directly access attached drives in JBOD mode rather than as RAID. If you can afford them, you can buy the LSI 9211-8i HBA card. Alternatively, you can also buy a less expensive card, and then replace its stock "IR" (Initiator-RAID) firmware by "crossflashing" to an "IT" (Initiator-Target) version of LSI's general firmware for 9211-8i hardware. This option is useful for people building home NAS systems on a budget. Popular cards include the Dell PERC H200 and the IBM ServeRAID M1015. This ServeTheHome post introduces the topic well.

Here is the relevant dmesg for a Dell PERC H200 Internal (H200I) under FreeBSD 8.4-RELEASE. (Note that this particular card's LSI firmware (Phase 9) is out of sync with the FreeBSD driver (Phase 14), which may have unexpected side effects. The system was initially built as a FreeBSD 8.1-RELEASE system in 2010.)

$ uname -r
8.4-RELEASE-p19
$ egrep ^mps0 /var/run/dmesg.boot
mps0: <LSI SAS2008> port 0xc000-0xc0ff mem 0xfb3b0000-0xfb3bffff,0xfb3c0000-0xfb3fffff irq 16 at device 0.0 on pci3
mps0: Firmware: 09.00.00.00, Driver: 14.00.00.01-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: [ITHREAD]

General flashing tips

Before flashing, and especially before erasing any flash, use the sas2flsh.exe -listall option to note the SAS ID of your device (usually beginning with "0x590"). If you accidentally erase the entire flash (sas2flsh.exe -o -e 6 will retain your SAS ID, but sas2flsh.exe -o -e 7 will wipe it), you will not be able to re-flash the device unless you have this ID. Write it down.

Some earlier versions of sas2flsh.exe allow cards to be flashed from IR firmware to IT firmware; others do not. I and others have had luck with the one that comes with LSI's Phase 7 (AKA P7 or P07) firmware. (Try this link, or search LSI.com for "9211_8i_Package_For_P7_Firmware_BIOS_Upgrade_on_MSDOS_and_Windows" to download the package that contains this version of sas2flsh.exe.

To flash the firmware on cards installed in non-UEFI motherboards, you can create a DOS-bootable USB key using a tool like Rufus. Rufus will make the device bootable with FreeDOS or MS-DOS (well, actually, Windows ME!). I and others have had better luck using the MS-DOS option. (According to that thread, LSI themselves recommend MS-DOS rather than FreeDOS).

Also note that when flashing using sas2flsh.exe there are two different components to be flashed: the firmware (contained in a filename sometimes ending with .fw, and usually named after the device in some way) and the BIOS (usually named something like MPTSAS2.ROM). The firmware component is what your OS driver communicates with. The BIOS component allows you to configure the firmware at boot time, and can enumerate the list of attached hard drives. For ZFS and JBOD purposes, the BIOS is not strictly necessary, and has even been reported to cause problems when present. Erasing the firmware areas sas2flsh.exe -o -e 6 and then just applying the firmware without the BIOS will also result in faster boot times.

A common error that people get when flashing is "Failed to Validate Mfg Page 2". This occurs when you try to flash to the LSI firmware without first erasing the firmware. The techmattr blog has some good information.

Phase 10 firmware or higher is needed for cards in this family (6GB/s HBSa) in order to support drives larger than 2GB. See this LSI KB article (old version cached at the Internet Archive)

FreeBSD flashing considerations

At this writing (2015-01), there have been reports of Phase 20 not playing well with FreeNAS and FreeBSD. Downgrading to Phase 16 (FreeBSD 9.3 and 10.0) or Phase 19 (FreeBSD 10.1) is reported to be more stable.

Under FreeBSD, PC-BSD, and FreeNAS, the desired end state is for the "Firmware" and "Driver" ports of the dmesg line to use identical firmware versions. For FreeBSD 10.1-RELEASE, this is the Phase 19 version. In the dmesg output, the Firmware item is what's on the card, and the Driver item is what the OS supplies.

mps0: Firmware: 19.00.00.00, Driver: 19.00.00.00-fbsd

In fact, FreeNAS will even complain if they are mismatched.

(I also list all of the firmware/OS pairings I know of towards the end of this post.)

Beware when upgrading a FreeBSD-based OS. Depending on the combination of firmware and driver, your drives may disappear from the OS' view until you reflash. This can be especially troublesome if your root filesystem is ZFS.

How to reflash the Dell Internal Tape Adapter 15MCV card as a 9211-8i

There is a card from Dell that looks almost identical to the H200I card, but is actually a Dell Internal Tape Adapter board (Dell part number 15MCV). This is identified in various levels of firmware and utilities as "Int Tape Adapter" or "IntTapeAdptr", and identified under Linux as:

 Vendor(0x1000), Device(0x0072), SSVID(0x1028), SSDID(0x1F22)

Cards labeled as "H200" on eBay are sometimes actually these cards instead. Unfortunately, the usual methods for flashing to generic LSI drivers do not work for the Tape Adapter boards. But as discovered by Hardforum user lamune in this post, if you start from the original Dell Internal Tape Adapter firmware, and then, without erasing the current firmware, flash using Supermicro HBA drivers (Phase 16 at this writing) as an intermediate step, you can then flash to the LSI firmware.

Here is the Linux dmesg for my Internal Tape Adapter board, prior to being cross-flashed. Note that capabilities include RAID, and the BIOS has a standard version (07.11.10.00):

$ dmesg | egrep -i 'lsi|mpt|mps|sas'
[    0.000000]   HighMem  empty
[    5.722377] mpt2sas version 16.100.00.00 loaded
[    5.731608] scsi4 : Fusion MPT SAS Host
[    5.739575] mpt2sas0: 32 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (497212 kB)
[    5.739643] mpt2sas 0000:01:00.0: irq 43 for MSI/MSI-X
[    5.739682] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 43
[    5.739686] mpt2sas0: iomem(0x00000000dfcb0000), mapped(0xe0280000), size(65536)
[    5.739689] mpt2sas0: ioport(0x000000000000dc00), size(256)
[    6.028016] mpt2sas0: sending diag reset !!
[    7.268013] mpt2sas0: diag reset: SUCCESS
[    7.418064] mpt2sas0: Allocated physical memory: size(4134 kB)
[    7.418070] mpt2sas0: Current Controller Queue Depth(2748), Max Controller Queue Depth(2879)
[    7.418073] mpt2sas0: Scatter Gather Elements per IO(128)
[    7.648484] mpt2sas0: LSISAS2008: FWVersion(07.15.08.00), ChipRevision(0x03), BiosVersion(07.11.10.00)
[    7.648490] mpt2sas0: Dell 6Gbps SAS: Vendor(0x1000), Device(0x0072), SSVID(0x1028), SSDID(0x1F22)
[    7.648492] mpt2sas0: Protocol=(Initiator,Target), Capabilities=(Raid,TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[    7.648577] mpt2sas0: sending port enable !!
[   10.168254] mpt2sas0: host_add: handle(0x0001), sas_addr(0x590bxxxxxxxxxxxx), phys(8)
[   15.296010] mpt2sas0: port enable: SUCCESS

Here is a Linux dmesg after successful crossflash of firmware, but skipping installing a BIOS. Note that capabilities no longer include RAID, and BIOS is empty (00.00.00.00)

$ dmesg | egrep -i 'lsi|mpt|mps|sas'
[    0.000000]   HighMem  empty
[    5.784639] mpt2sas version 16.100.00.00 loaded
[    5.789191] scsi4 : Fusion MPT SAS Host
[    5.793970] mpt2sas0: 32 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (497212 kB)
[    5.794039] mpt2sas 0000:01:00.0: irq 43 for MSI/MSI-X
[    5.794081] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 43
[    5.794086] mpt2sas0: iomem(0x00000000dfcb0000), mapped(0xe0140000), size(65536)
[    5.794088] mpt2sas0: ioport(0x000000000000dc00), size(256)
[    6.235645] mpt2sas0: Allocated physical memory: size(4964 kB)
[    6.235652] mpt2sas0: Current Controller Queue Depth(3307), Max Controller Queue Depth(3432)
[    6.235654] mpt2sas0: Scatter Gather Elements per IO(128)
[    6.468421] mpt2sas0: LSISAS2008: FWVersion(19.00.00.00), ChipRevision(0x03), BiosVersion(00.00.00.00)
[    6.468429] mpt2sas0: Dell 6Gbps SAS: Vendor(0x1000), Device(0x0072), SSVID(0x1028), SSDID(0x1F22)
[    6.468432] mpt2sas0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[    6.468517] mpt2sas0: sending port enable !!
[    8.978905] mpt2sas0: host_add: handle(0x0001), sas_addr(0x590b11c017d2a400), phys(8)
[   14.116010] mpt2sas0: port enable: SUCCESS

Known FreeBSD versions and their equivalent target mps driver versions

  • 8.2-RELEASE: Phase 12? - not sure, but likely 12.00.00.00-fbsd - did not ship with, but it can be backported
  • 8.3-RELEASE: Phase 13 - mps0: Firmware: xx.xx.xx.xx, Driver: 13.00.00.00-fbsd
  • 8.4-RELEASE: Phase 14 - mps0: Firmware: xx.xx.xx.xx, Driver: 14.00.00.01-fbsd - LSI P14 firmware
  • 9.1-RELEASE: Phase 14 - mps0: Firmware: xx.xx.xx.xx, Driver: 14.00.00.01-fbsd
  • 9.2-RELEASE: Phase 14 - mps0: Firmware: xx.xx.xx.xx, Driver: 14.00.00.01-fbsd
  • FreeNAS v ?? Phase 15 - mps0: Firmware: xx.xx.xx.xx, Driver: 15.00.00.00-fbsd - LSI P15 firmware (ref)
  • 9.3-RELEASE: Phase 16 - mps0: Firmware: xx.xx.xx.xx, Driver: 16.00.00.00-fbsd - LSI P16 firmware
  • 10.0-RELEASE: Phase 16 - mps0: Firmware: xx.xx.xx.xx, Driver: 16.00.00.00-fbsd
  • Phase 18 was committed but not in a release that I can tell.
  • 10.1-RELEASE: Phase 19 - mps0: Firmware: xx.xx.xx.xx, Driver: 19.00.00.00-fbsd - LSI P19 firmware
  • 10.2-BETA2: Phase 20 - mps0: Firmware: xx.xx.xx.xx, Driver: 20.00.00.00-fbsd - LSI P20 firmware (reported by Dan Langille)
  • 10.3-RELEASE: Phase 20 - mps0: Firmware: xx.xx.xx.xx, Driver: 20.00.00.00-fbsd - LSI P20 firmware
  • 11.0-RELEASE: Phase 20? - mps0: Firmware: 20.00.07.00, Driver: 21.01.00.00-fbsd - (The FreeBSD driver is version 21, but the latest Avago firmware download (from 2016-04) still shows Phase 20 as the most recent).

Note:After Avago bought LSI, their new download system sometimes makes direct linking more difficult. This search may help.

FreeBSD firmware installers

I haven't had good luck with these, because they often can only perform a small subset of the actions necessary to upgrade firmware. But if you need them, here they are.

(For all firmware and installers, see LSI's archive)

Useful command-line snippets

sas2flsh -listall
sas2flsh -c 0 -o -testssid 1028:%SSID% > SSID.out
sas2flsh -c %num% -f fwname.fw > flash.out
sas2flsh -c %num% -b mptsas2.rom >> flash.out
sas2flsh -c %num% -b x64sas2.rom >> flash.out
sas2flsh -c %num% -o -reset > reset.out

References

Posted by tychotithonus (Royce Williams) at 2:43 PM
Labels: , ,
blog comments powered by Disqus