Sean's Blog

We’re building out a few new datacenters at work now, and before I can get a start on the Linux infrastructure, we’ve been working on the core infrastructure (network and SAN). We’re fortunate in that we’re able to use some of the latest-and-greatest technologies, including some nice new 10GigE switches that support FCoE, along with necessary CNAs (converged network adapters).

What makes these really cool is your NICs and HBAs are now one in the same. For a given level of redundancy and bandwidth, your port count is cut in half. And with these fancy Twinax cables/transceivers you can get, you don’t need to use fiber to leverage these benefits.

Of course, nothing’s perfect. I haven’t been working on the networking side of things too much, but it looks like it’s been quite a pain to architect the design of what will be a truly converged network, with a pair of switches providing redundant network and fabric connectivity to the SAN (requiring things like VPCs, etc.).

Among the things I’ve noticed is the difficulty of getting the NIC side of the CNA bonded on Linux. Let’s say you configure a switch from a certain vendor whose name rhymes with Crisco such that you have “channel-group XXX mode on” for each port. Great. You can boot from SAN just fine, and all the stuff on the FC side works just fine. But no networking. OK, you figure. Let’s try “channel-group XXX mode active” on each switch port, and then configure LACP on each host. Yeah, don’t do that. It’ll get networking going, but it’ll break FCoE connectivity, which especially sucks when your OS volume(s) are stored on the SAN.

As best as I can figure it, no CNA on the market today (and perhaps the FCoE standard itself) allows for LACP to be configured at the adapter level. Naturally, my first thought was to try active/passive bonding at the Linux level instead, but that didn’t work. Inexplicably, a vendor suggested we try XOR load balancing (mode=3 when you load the bonding module), and that managed to get things working.

As nice as it is to cut your necessary port count in half, it’s kind of scary when you’re putting everything (network and storage) in one bucket. Especially when the documentation hasn’t quite caught up to the technology.