Sunday, January 25, 2015

Managing and optimizing lists of password masks

I've been working on some password-cracking research on the side. I thought I'd come up with a cool new idea, but it turns out that someone else already thought of it.

It occurred to me last night that a big list of passwords could be abstracted out into their equivalent masks, and then a frequency count of those masks could be generated, which could then be exhausted in frequency order.

First, I extracted a frequency count of character set combinations (masks) from all eight-characters-longthe RockYou breach's password list, yielding a list of the form:

100:hundredofthese
95: 95ofthese
[...]
2:justtwoofthese
1:onlyoneofthese
1:alsoonlyoneofthese

... as follows:

#!/bin/bash

echo "- Getting frequency of character patterns from RockYou ..."
time gunzip -cd rockyou.txt.gz \
        | tr '[:lower:]' 'l' \
        | tr '[:upper:]' 'u' \
        | tr '[:digit:]' 'd' \
        | tr "[\ !\"#$%amp;&\'()*+,-./:;<=>?@\[\\\]^_\`{|}~]" 's' \
        | sed 's/[^luds]/a/g' \
        | strings \
        | cut -b1-8 \
        | freqcount \
        > rockyou.freq.8a
wc -l rockyou.freq.8a
head rockyou.freq.8a

echo "- Generate masks."
echo "- Ignoring all masks with more than three consecutive 'a' charset."
time cat rockyou.freq.8a \
        | cut -d\: -f2 \
        | sed 's/l/?l/g;s/u/?u/g;s/d/?d/g;s/s/?s/g;s/a/?a/g' \
        | egrep -v 'aaaa' \
        > rockyou.masks.8
wc -l rockyou.masks.8
head rockyou.masks.8

echo "- Done."
#end of script

Next, I wrote a script to exhaust each one in order by frequency using hashcat:

#!/bin/bash

for mymask in `rockyou.masks.8`; do
        echo "- Running mask: $mymask ..."
        cudaHashcat64.bin -a 3 -m 1500 \
                target-hashes.list \
                $mymask
        echo "$mymask: done - `date`" >> $0.log
done
#end of script

Then it occurred to me that if someone else had published this info, and had used real corpora of passwords as the input, then our frequency lists would probably look similar. So I did the following Google search:

"?l?l?l?d?d?d?d" "?l?l?l?l?l?d?d?d"

... and the first hit was the KoreLogic blog post.

Dangit! :-) But at least I'm catching up to the state of the art; the KoreLogic article was published in April 2014. :-)

I got the idea from work I had done on some license-plate-collecting stuff I do on the side. I thought of it for capturing high-level patterns in serials, so that people can search for a plate based on the serial. A plate with "BDT 606" on it would match any plate whose serial "mask" is "AAA 999" using my notation. (I then match more closely, but it's used for a high-level search first).

I haven't watched the KoreLogic presentation yet, but I can definitely improve upon my own approach, because I'm being overly aggressive in turning then entire set of non-alphanumeric-but-printable characters into 's':

        | tr "[\ !\"#$%&\'()*+,-./:;<=>?@\[\\\]^_\`{|}~]" 's' \

... when most folks use the simple ones (#$%@, etc.) I could create a custom charset for this using the notation as noted here ... and then turn the remaining characters into another custom charset that is the remaining characters.

I then found PACK - the Password Analysis and Cracking Kit, which is is a set of Python scripts to manage masks, including optimizing a set of masks based on a given timeframe (or, "I have 24 hours. Which masks should I use to maximize how many passwords I can crack?")

Posted by tychotithonus (Royce Williams) at 2:52 PM View Comments

FreeBSD LSI SAS9211-8i HBA firmware notes

I'll be using this post to store information about LSI HBA firmware, with a focus on FreeBSD (but also drawing upon Linux information). It may also be useful for users of FreeNAS, PC-BSD, unRAID, Nexenta, or ZFSguru.

Why - SATA port density on a budget

If you are using ZFS, you do not need RAID -- you just need lots of fast SATA ports. To maximize the features of ZFS, it needs to directly access attached drives in JBOD mode rather than as RAID. If you can afford them, you can buy the LSI 9211-8i HBA card. Alternatively, you can also buy a less expensive card, and then replace its stock "IR" (Initiator-RAID) firmware by "crossflashing" to an "IT" (Initiator-Target) version of LSI's general firmware for 9211-8i hardware. This option is useful for people building home NAS systems on a budget. Popular cards include the Dell PERC H200 and the IBM ServeRAID M1015. This ServeTheHome post introduces the topic well.

Here is the relevant dmesg for a Dell PERC H200 Internal (H200I) under FreeBSD 8.4-RELEASE. (Note that this particular card's LSI firmware (Phase 9) is out of sync with the FreeBSD driver (Phase 14), which may have unexpected side effects. The system was initially built as a FreeBSD 8.1-RELEASE system in 2010.)

$ uname -r
8.4-RELEASE-p19
$ egrep ^mps0 /var/run/dmesg.boot
mps0: <LSI SAS2008> port 0xc000-0xc0ff mem 0xfb3b0000-0xfb3bffff,0xfb3c0000-0xfb3fffff irq 16 at device 0.0 on pci3
mps0: Firmware: 09.00.00.00, Driver: 14.00.00.01-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: [ITHREAD]

General flashing tips

Before flashing, and especially before erasing any flash, use the sas2flsh.exe -listall option to note the SAS ID of your device (usually beginning with "0x590"). If you accidentally erase the entire flash (sas2flsh.exe -o -e 6 will retain your SAS ID, but sas2flsh.exe -o -e 7 will wipe it), you will not be able to re-flash the device unless you have this ID. Write it down.

Some earlier versions of sas2flsh.exe allow cards to be flashed from IR firmware to IT firmware; others do not. I and others have had luck with the one that comes with LSI's Phase 7 (AKA P7 or P07) firmware. (Try this link, or search LSI.com for "9211_8i_Package_For_P7_Firmware_BIOS_Upgrade_on_MSDOS_and_Windows" to download the package that contains this version of sas2flsh.exe.

To flash the firmware on cards installed in non-UEFI motherboards, you can create a DOS-bootable USB key using a tool like Rufus. Rufus will make the device bootable with FreeDOS or MS-DOS (well, actually, Windows ME!). I and others have had better luck using the MS-DOS option. (According to that thread, LSI themselves recommend MS-DOS rather than FreeDOS).

Also note that when flashing using sas2flsh.exe there are two different components to be flashed: the firmware (contained in a filename sometimes ending with .fw, and usually named after the device in some way) and the BIOS (usually named something like MPTSAS2.ROM). The firmware component is what your OS driver communicates with. The BIOS component allows you to configure the firmware at boot time, and can enumerate the list of attached hard drives. For ZFS and JBOD purposes, the BIOS is not strictly necessary, and has even been reported to cause problems when present. Erasing the firmware areas sas2flsh.exe -o -e 6 and then just applying the firmware without the BIOS will also result in faster boot times.

A common error that people get when flashing is "Failed to Validate Mfg Page 2". This occurs when you try to flash to the LSI firmware without first erasing the firmware. The techmattr blog has some good information.

Phase 10 firmware or higher is needed for cards in this family (6GB/s HBSa) in order to support drives larger than 2GB. See this LSI KB article (old version cached at the Internet Archive)

FreeBSD flashing considerations

At this writing (2015-01), there have been reports of Phase 20 not playing well with FreeNAS and FreeBSD. Downgrading to Phase 16 (FreeBSD 9.3 and 10.0) or Phase 19 (FreeBSD 10.1) is reported to be more stable.

Under FreeBSD, PC-BSD, and FreeNAS, the desired end state is for the "Firmware" and "Driver" ports of the dmesg line to use identical firmware versions. For FreeBSD 10.1-RELEASE, this is the Phase 19 version. In the dmesg output, the Firmware item is what's on the card, and the Driver item is what the OS supplies.

mps0: Firmware: 19.00.00.00, Driver: 19.00.00.00-fbsd

In fact, FreeNAS will even complain if they are mismatched.

(I also list all of the firmware/OS pairings I know of towards the end of this post.)

Beware when upgrading a FreeBSD-based OS. Depending on the combination of firmware and driver, your drives may disappear from the OS' view until you reflash. This can be especially troublesome if your root filesystem is ZFS.

How to reflash the Dell Internal Tape Adapter 15MCV card as a 9211-8i

There is a card from Dell that looks almost identical to the H200I card, but is actually a Dell Internal Tape Adapter board (Dell part number 15MCV). This is identified in various levels of firmware and utilities as "Int Tape Adapter" or "IntTapeAdptr", and identified under Linux as:

 Vendor(0x1000), Device(0x0072), SSVID(0x1028), SSDID(0x1F22)

Cards labeled as "H200" on eBay are sometimes actually these cards instead. Unfortunately, the usual methods for flashing to generic LSI drivers do not work for the Tape Adapter boards. But as discovered by Hardforum user lamune in this post, if you start from the original Dell Internal Tape Adapter firmware, and then, without erasing the current firmware, flash using Supermicro HBA drivers (Phase 16 at this writing) as an intermediate step, you can then flash to the LSI firmware.

Here is the Linux dmesg for my Internal Tape Adapter board, prior to being cross-flashed. Note that capabilities include RAID, and the BIOS has a standard version (07.11.10.00):

$ dmesg | egrep -i 'lsi|mpt|mps|sas'
[    0.000000]   HighMem  empty
[    5.722377] mpt2sas version 16.100.00.00 loaded
[    5.731608] scsi4 : Fusion MPT SAS Host
[    5.739575] mpt2sas0: 32 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (497212 kB)
[    5.739643] mpt2sas 0000:01:00.0: irq 43 for MSI/MSI-X
[    5.739682] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 43
[    5.739686] mpt2sas0: iomem(0x00000000dfcb0000), mapped(0xe0280000), size(65536)
[    5.739689] mpt2sas0: ioport(0x000000000000dc00), size(256)
[    6.028016] mpt2sas0: sending diag reset !!
[    7.268013] mpt2sas0: diag reset: SUCCESS
[    7.418064] mpt2sas0: Allocated physical memory: size(4134 kB)
[    7.418070] mpt2sas0: Current Controller Queue Depth(2748), Max Controller Queue Depth(2879)
[    7.418073] mpt2sas0: Scatter Gather Elements per IO(128)
[    7.648484] mpt2sas0: LSISAS2008: FWVersion(07.15.08.00), ChipRevision(0x03), BiosVersion(07.11.10.00)
[    7.648490] mpt2sas0: Dell 6Gbps SAS: Vendor(0x1000), Device(0x0072), SSVID(0x1028), SSDID(0x1F22)
[    7.648492] mpt2sas0: Protocol=(Initiator,Target), Capabilities=(Raid,TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[    7.648577] mpt2sas0: sending port enable !!
[   10.168254] mpt2sas0: host_add: handle(0x0001), sas_addr(0x590bxxxxxxxxxxxx), phys(8)
[   15.296010] mpt2sas0: port enable: SUCCESS

Here is a Linux dmesg after successful crossflash of firmware, but skipping installing a BIOS. Note that capabilities no longer include RAID, and BIOS is empty (00.00.00.00)

$ dmesg | egrep -i 'lsi|mpt|mps|sas'
[    0.000000]   HighMem  empty
[    5.784639] mpt2sas version 16.100.00.00 loaded
[    5.789191] scsi4 : Fusion MPT SAS Host
[    5.793970] mpt2sas0: 32 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (497212 kB)
[    5.794039] mpt2sas 0000:01:00.0: irq 43 for MSI/MSI-X
[    5.794081] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 43
[    5.794086] mpt2sas0: iomem(0x00000000dfcb0000), mapped(0xe0140000), size(65536)
[    5.794088] mpt2sas0: ioport(0x000000000000dc00), size(256)
[    6.235645] mpt2sas0: Allocated physical memory: size(4964 kB)
[    6.235652] mpt2sas0: Current Controller Queue Depth(3307), Max Controller Queue Depth(3432)
[    6.235654] mpt2sas0: Scatter Gather Elements per IO(128)
[    6.468421] mpt2sas0: LSISAS2008: FWVersion(19.00.00.00), ChipRevision(0x03), BiosVersion(00.00.00.00)
[    6.468429] mpt2sas0: Dell 6Gbps SAS: Vendor(0x1000), Device(0x0072), SSVID(0x1028), SSDID(0x1F22)
[    6.468432] mpt2sas0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[    6.468517] mpt2sas0: sending port enable !!
[    8.978905] mpt2sas0: host_add: handle(0x0001), sas_addr(0x590b11c017d2a400), phys(8)
[   14.116010] mpt2sas0: port enable: SUCCESS

Known FreeBSD versions and their equivalent target mps driver versions

  • 8.2-RELEASE: Phase 12? - not sure, but likely 12.00.00.00-fbsd - did not ship with, but it can be backported
  • 8.3-RELEASE: Phase 13 - mps0: Firmware: xx.xx.xx.xx, Driver: 13.00.00.00-fbsd
  • 8.4-RELEASE: Phase 14 - mps0: Firmware: xx.xx.xx.xx, Driver: 14.00.00.01-fbsd - LSI P14 firmware
  • 9.1-RELEASE: Phase 14 - mps0: Firmware: xx.xx.xx.xx, Driver: 14.00.00.01-fbsd
  • 9.2-RELEASE: Phase 14 - mps0: Firmware: xx.xx.xx.xx, Driver: 14.00.00.01-fbsd
  • FreeNAS v ?? Phase 15 - mps0: Firmware: xx.xx.xx.xx, Driver: 15.00.00.00-fbsd - LSI P15 firmware (ref)
  • 9.3-RELEASE: Phase 16 - mps0: Firmware: xx.xx.xx.xx, Driver: 16.00.00.00-fbsd - LSI P16 firmware
  • 10.0-RELEASE: Phase 16 - mps0: Firmware: xx.xx.xx.xx, Driver: 16.00.00.00-fbsd
  • Phase 18 was committed but not in a release that I can tell.
  • 10.1-RELEASE: Phase 19 - mps0: Firmware: xx.xx.xx.xx, Driver: 19.00.00.00-fbsd - LSI P19 firmware
  • 10.2-BETA2: Phase 20 - mps0: Firmware: xx.xx.xx.xx, Driver: 20.00.00.00-fbsd - LSI P20 firmware (reported by Dan Langille)
  • 10.3-RELEASE: Phase 20 - mps0: Firmware: xx.xx.xx.xx, Driver: 20.00.00.00-fbsd - LSI P20 firmware
  • 11.0-RELEASE: Phase 20? - mps0: Firmware: 20.00.07.00, Driver: 21.01.00.00-fbsd - (The FreeBSD driver is version 21, but the latest Avago firmware download (from 2016-04) still shows Phase 20 as the most recent).

Note:After Avago bought LSI, their new download system sometimes makes direct linking more difficult. This search may help.

FreeBSD firmware installers

I haven't had good luck with these, because they often can only perform a small subset of the actions necessary to upgrade firmware. But if you need them, here they are.

(For all firmware and installers, see LSI's archive)

Useful command-line snippets

sas2flsh -listall
sas2flsh -c 0 -o -testssid 1028:%SSID% > SSID.out
sas2flsh -c %num% -f fwname.fw > flash.out
sas2flsh -c %num% -b mptsas2.rom >> flash.out
sas2flsh -c %num% -b x64sas2.rom >> flash.out
sas2flsh -c %num% -o -reset > reset.out

References

Posted by tychotithonus (Royce Williams) at 2:43 PM View Comments
Labels: , ,
Subscribe to: Posts (Atom)