Home Lab 10Gbit Adventures… (X520-DA2 EEPROM)

April 26th, 2020 Posted in General

So I happened to have some Intel X520-DA2 ethernet adapters in a box in my basement that came from deprecated NetApp filers. I’d used these with FreeBSD/FreeNAS in the past and they worked as expected aside from the fact that the ports would alternate which ones were enabled. If I plugged in ix0 when I rebooted it would be dead and I would have to use ix1 and vice versa. Needless to say this annoyed the crap out of me. I bandaided the problem by setting up an active/passive bond of both ports but LAGG combined with VLAN tags and Jumbo Frames resulted in weird stability issues so I abandoned the whole adventure and just used 1Gbit for a long time.

A few weeks ago I decided to dump FreeNAS in favor of ZFS-on-linux with the intention of running docker containers natively and avoiding the Bhyve layer that (I believe) just slowed things down. In the process I stuffed my X520-DA2 card back in the NAS. It wouldn’t work! Not with twinax, not with Intel SFP+ modules, no matter what I did i’d just get NO-CARRIER.

Then I started exploring the EEPROM. Knowing about this post to get it to support any optics you wanted, I also stumbled across this additional post which was to get the card to work in an Apple Mac Pro.

Changing the following setting of my card back to what appeared to be the default subsystem ID based on this posters default settings of 7a11 vs my current 000c.

ethtool -E enp4s0f0 magic 0x10fb8086 offset 0x32a value 0x11
ethtool -E enp4s0f0 magic 0x10fb8086 offset 0x32b value 0x7a
ethtool -E enp4s0f1 magic 0x10fb8086 offset 0x32a value 0x11
ethtool -E enp4s0f1 magic 0x10fb8086 offset 0x32b value 0x7a

And to validate…
root@server0:~# ethtool -e enp4s0f0 | grep 0x032
0x0320:         03 00 1f 00 00 00 00 2b 03 13 11 7a 86 80 a6 10 
root@server0:~# ethtool -e enp4s0f1 | grep 0x032
0x0320:         03 00 1f 00 00 00 00 2b 03 13 11 7a 86 80 a6 10 

This got me 50% of the way there. The cards worked with twinax, but SFP+ was still a no go! Rewinding to the first post at Serve The Home about SFP+ optics I noticed an addition setting in 0x50 was different. At 0x5c I had 0b but the original poster had 11. So I tried the following.

ethtool -E enp4s0f0 magic 0x10fb8086 offset 0x5c value 0x11
ethtool -E enp4s0f1 magic 0x10fb8086 offset 0x5c value 0x11

And validate again…
root@server0:~# ethtool -e enp4s0f0 | grep 0x005
0x0050:         0a 0e 93 00 ff ff bc 0c fc ff 01 00 11 5e ff ff 
root@server0:~# ethtool -e enp4s0f1 | grep 0x005
0x0050:         0a 0e 93 00 ff ff bc 0c fc ff 01 00 11 5e ff ff 

VIOLA! Now the cards work with both twinax and SFP+ modules of any brand!

I would imagine that changing these settings back to my originals would allow an Intel OEM card to work in a NetApp filer as well, but some other values that I’ve not yet explored may be different.

Post a Comment