Transcript
And continuing with our dev and home lab theme, we're eating from the trash again. This is a ConnectX-3 that has built-in LSI SAS connections. Recently did a video on ZFS high availability, and this card is oddly well-suited to be able to do that. It's Data Direct Networks. These things are not even listed correctly on eBay. They're typically $10 to $35. I don't think one of these cards is worth more than about 50 bucks, if that. These are ancient cards, and they can be good for the home lab cuz 40 gigabit plus 4 * 4 serial attached SCSI external ports. Uh but there's some hoops you're going to jump through, so let's discuss.
All right, so this is a PCIe Gen3 by 8 interface. And it has a a PLX, technically a PEX, bridge for more PCIe devices. And there are two LSI SAS controllers on here, which generally up till now people have not been able to get to work, and then a Mellanox ConnectX-3. Now, the Mellanox ConnectX-3, there's two variants of this card. You can get a 40 gig InfiniBand, or you can get 40 gig Ethernet. And you can cross-flash, mostly. Uh there's There are reports of successful cross-flashing on the Mellanox side, but really you're only guaranteed, you know, like the 10 gigabit version of that. But the SAS controllers have not really worked up till now.
So, I started to work on that to see if that's something that I could fix. Actually, Alan Jude turned me onto this at the 45 Drives Creator Summit. Yeah, I was just in Wilmington taking a look at their new stuff. 45 Drives has productized, you remember the uh B70 SR-IOV? If you would like a VDI solution from 45 Drives that's based around the Intel Arc B70, so there's no monthly subscription, they productized it. That's what I I showed that off last year, and they were like, "Yeah, you showed that off last year. Now selling a product that's based on that. So, you want Proxmox plus a B70, it's turnkey, it works really well.
Also, with Chasm workspaces, the hardware, you can just set it up with Chasm directly. I met the Chasm workspaces guy, that was a lot of fun. But, this is about eating from the trash in terms of, you know, hardware cuz this is the thing that we find ourselves in. This also really hurts my soul because I've made fun of 40 gigabit Ethernet. And you have to understand, like 10 gigabit / 40 gigabit is different than 10 gigabit / 25 gigabit. The 10 / 25 gigabit is a modern standard with reasonable power consumption and a reasonable design. This is the old design. It's 10 / 40 gigabit. It's like different error correction, like I don't know.
It's like PAM versus like not. It's a whole conversation. I don't even want to get into it think about. This has been obsolete since I've been encouraging you to build out your home lab to be faster than 10 gigabit. But, oh, in the climate that we find ourselves in, uh, we're we're pulling e-waste back out of the the dumpster. But, something else is happening with AI. Like AI as a usable tool for helping with diagnostics and reverse engineering mean that hardware like this, which was, you know, previously unreachable, is usable. Because in the past, this thing really only worked as a ConnectX-3.
The SAS wouldn't initialize. And the reason the SAS wouldn't initialize is because there's not really a flash ROM on here. And so, like booting up the SAS controller and getting the disks to be detected didn't really work on on this controller properly. So, anyway, fast forward, I wrote a guide on the Level One Techs forum, which uses another guy's utility that goes all the way back to like recovering from a bad flash on an SLSI controller. Except you can do something called host booting. So, you can boot these controllers off of the host, and that works fine. The IOMMU trick is you can go into the PCIE bridge for this thing and set some set some parameters, basically.
And uh, the IO MMU problem becomes nonexistent. So, you can still use IO MMU, you can still use you know, IO MMU isolation, and you have two eight-port SAS controllers behind this PCIe bridge, which is great. So, two of your four connectors here are it's it's like a 9205. You got one 9205, a second 9205, plus a Mellanox ConnectX-3, three different distinct peripherals on one PCIe card. That's about 35 bucks. And you're going to need to how to if you want to use the the LSI side of it, the ConnectX part of it basically works as advertised. But for a single PCIe slot, you can get a pretty fast network connection.
That's in the how to the forum forum thread. When you're getting these from eBay, they don't know which one it is. You're get you're getting a Data Direct Networks uh 00232, you know, -302 rev 2 or 04-00232. Some of the eBay listings call this a video card. But listen, if it's like 8 to 35 dollars, don't correct them. Don't interrupt your enemy, the person pricing that, when they're making a mistake. Just yeah. Also, the 10 gigabit / 40 gigabit Ethernet switches, sometimes you can get some of those that are pretty inexpensive, and that might make a reasonable backbone for your home lab. This card runs hot and needs direct air flow.
It uses 37 watts of power when it's idle. So, the floor on power consumption on your system that features one or two of these cards is going to be 37 watts per card. It's a lot of power. And I've had a little bit more time with the DDN card. I have some bad news. You can't do 40 gig Ethernet with it. At least I've really tried a lot of reverse engineering. I tried pulling out all the stops. I even got our robot helpers to to help. I've got the the Dell XPS 16, which has, you know, 128 gigs of up to 128 gigs of of local memory running some some AI models, the XP XPS 16 creator Edition. And I've been trying some other, you know, it's like look at the Yeah, because it the CX3 is pretty well understood at this point.
There's different PS IDs and IN Is. There's a lot of structural or architectural things that are really well known about these Mellanox ConnectX-3 cards and how you can you know, sort of modify them a little bit from factory. Maybe you flash a Dell or an HP firmware image on a vanilla one and you get a little more functionality because, you know, those OEMs use those chips for a little longer than than anybody else. I couldn't get 40 gig Ethernet to work on this PHY. 10 gigabit Ethernet is pretty reliable and 40 gig Infiniband is pretty reliable, but uh it's a that's the best I can do. And fixing up IOMMU so you don't have to disable IOMMU and there's a full thread on the forum that has all of the details and and the full write-up.
And um yeah, I don't know. This is this is just another fun example of our super unlocked amazing future with local AI because, you know, Anthropic's model starting to refuse to do this kind of stuff. Oh, you're using that in a way that the manufacturer didn't intend. That's my God-given right as an American. But I digress. These SAS connections, SAS 2, it's not super fast. It's a 600 megabit per channel, four channels per connector, four connectors on the card. So, that's what? 2.4 gigabit per connection times four connections. It's not bad. It's not It's fine for mechanical hard drives. And it's fine for home lab.
And this is a a card that I also used as part of subsequent ZFS high availability testing. I didn't use this card initially, but I have been using it um since discovering it. Thanks, Allan Jude. Uh with the with the disk shelf. So, yeah, one PCIe slot with networking and and SAS and also like hacking things and and that sort of stuff. If it sounds too complicated for you to set up, it's like configuring the kernel or whatever, you could literally take my how-to and ask OpenAI or Claude to help you set it up on your Linux box and it can. That's one of the test things that I do is like I take the thing and I try to run it through the AI to see if my instructions are clear enough for an AI to implement what I have written and then adjust it if not to help you do things, which is also kind of a theme.
I'm working on a on a a project have local offline AI that will run in 16 GB of VRAM or less that can help you configure your home lab and do stuff with your home lab including hack your TV and set up drivers and follow Level One Tech Forum how-tos. A lot of exciting things happening. I'm Wendell this Level One. This has been a a quickie quickie look at how we eat from the garbage. If there's other hardware that is only lacking software support out there that would be good for the home lab or would be good to reach out of the the dumpster and press back into service, let's do it because that's also going to be the subject of another video in general like more more grandiose, more cerebral, less less brass tacks, you know, man on the street.
But hey, I'm Wendell this Level One. This is why we have a forum. I'm signing out and I'll see you there.