forked from hswaw/hscloud
90 lines
3.3 KiB
Markdown
90 lines
3.3 KiB
Markdown
hbj11 - the Hackerspace Blade JBOD adapter
|
|
===
|
|
|
|
Replacement cards for M610 blades at the hackerspace.
|
|
|
|
|
|
|
|
Usually, Dell M610 blades have the following disk controller setup:
|
|
|
|
```
|
|
“Mezzanine connector”
|
|
.'-------'-------'.
|
|
.-----------------. .----------------.
|
|
| Motherboard | | Dell CERC card |
|
|
|-----------------| PCIe |----------------|
|
|
| |-------| PCIe-- | LSI |
|
|
| | SAS | .--| Crap |
|
|
| .-----|-------| SAS-' '-------|
|
|
| | | '----------------'
|
|
| | |
|
|
| | | SAS .----------------.
|
|
| '-----|-------| Disk Backplane |
|
|
'-----------------' '----------------'
|
|
| |
|
|
.-----. .-----.
|
|
| HDD | | HDD |
|
|
| | | |
|
|
| | | |
|
|
'-----' '-----'
|
|
```
|
|
|
|
However, the Dell CERCs are hot garbage. Put them in writeback mode, and the
|
|
non-battery-backed cache will lose data on power loss (and block the boot
|
|
process after reboot to let you know it lost data). Put it in write-through
|
|
mode, and lose most of your IOPS.
|
|
|
|
Turns out, the connector for the Dell CERC mezzanine cards carries not just
|
|
PCIe to the CPU's root complex and SAS to the disks, it also carries two SATA
|
|
lanes (non-SAS) to the built-in AHCI controller on the motherboard. Thus, we
|
|
can fabricate a 'jumper' card like this:
|
|
|
|
|
|
```
|
|
“Mezzanine connector”
|
|
.'-------'-------'.
|
|
.-----------------. .----------------.
|
|
| Motherboard | | HBJ11 |
|
|
|-----------------| PCIe |----------------|
|
|
| |-------| N/C |
|
|
| | SATA | |
|
|
| AHCI |-------|--. |
|
|
| | SAS | | |
|
|
| .-----|-------|--' |
|
|
| | | '----------------'
|
|
| | |
|
|
| | | SAS .----------------.
|
|
| '-----|-------| Disk Backplane |
|
|
'-----------------' '----------------'
|
|
| |
|
|
.-----. .-----.
|
|
| HDD | | HDD |
|
|
| | | |
|
|
| | | |
|
|
'-----' '-----'
|
|
```
|
|
|
|
This allows us to use the drives via the built in AHCI controller on the
|
|
motherboard! Of course, only one lane of each SAS lane is connected, so you can
|
|
only use SATA drives. But who cares about SAS, anyway. We just put SSDs in
|
|
these machines anyway.
|
|
|
|
There actually is a commercially available card from Dell that does almost
|
|
this, but it uses some expensive SATA redrivers. Turns out, you don't need them
|
|
- and we've tested this quite extensively.
|
|
|
|
There's one more thing on the connector to the card: an I2C flash that
|
|
identifies the card according to the IPMI FRU Information Storage Definition.
|
|
We need to place these on our custom cards and flash them. We decided to
|
|
over-engineer that, find that below.
|
|
|
|
Hardware
|
|
--------
|
|
|
|
A JLCSMT-compatible design for a four-layer 'HBJ11' jumper card is available in [hardware](hardware).
|
|
|
|
EEPROM & flasher
|
|
----------------
|
|
|
|
See [flasher](flasher) for a WebUSB/STM32 based flasher for the cards.
|