Hello,

I would like to connect a dual port SFP+ NetXtreme II BCM57711
10-Gigabit NIC to a HP J9309A ProCurve 4-Port 10GbE SFP+ zl Module
using a HP ProCurve 10-GbE SFP+ 7m Direct Attach Cable (J9285B).

Unfortuantely, it does not work. :( After connecting the switch and
the server together and loading the bnx2x module the switch logs:

I 05/20/10 18:32:23 ports: port E4 is Blocked by STP
I 05/20/10 18:32:23 ports: port E4 is now on-line

Here is the dmesg output from the server:

Broadcom NetXtreme II 5771x 10Gigabit Ethernet Driver bnx2x 1.52.1-5 (2009/11/09)
bnx2x 0000:04:00.0: PCI INT A -> GSI 38 (level, low) -> IRQ 38
bnx2x 0000:04:00.0: setting latency timer to 64
bnx2x: part number 394D4342-31373735-31314131-473331
bnx2x: Loading bnx2x-e1h-5.2.7.0.fw
bnx2x 0000:04:00.0: firmware: requesting bnx2x-e1h-5.2.7.0.fw
eth4: Broadcom NetXtreme II BCM57711 XGb (A0) PCI-E x8 5GHz (Gen2) found at mem dc000000, IRQ 38, node addr 00:10:18:5f:e4:b4
bnx2x 0000:04:00.1: PCI INT B -> GSI 45 (level, low) -> IRQ 45
bnx2x 0000:04:00.1: setting latency timer to 64
bnx2x: part number 394D4342-31373735-31314131-473331
bnx2x: Loading bnx2x-e1h-5.2.7.0.fw
bnx2x 0000:04:00.1: firmware: requesting bnx2x-e1h-5.2.7.0.fw
eth5: Broadcom NetXtreme II BCM57711 XGb (A0) PCI-E x8 5GHz (Gen2) found at mem dd000000, IRQ 45, node addr 00:10:18:5f:e4:b6
bnx2x 0000:04:00.0: irq 97 for MSI/MSI-X
bnx2x 0000:04:00.0: irq 98 for MSI/MSI-X
bnx2x 0000:04:00.0: irq 99 for MSI/MSI-X
bnx2x 0000:04:00.0: irq 100 for MSI/MSI-X
bnx2x 0000:04:00.0: irq 101 for MSI/MSI-X
bnx2x 0000:04:00.0: irq 102 for MSI/MSI-X
bnx2x: eth4: using MSI-X  IRQs: sp 97  fp[0] 99 ... fp[3] 102
ADDRCONF(NETDEV_UP): eth4: link is not ready
bnx2x 0000:04:00.1: irq 103 for MSI/MSI-X
bnx2x 0000:04:00.1: irq 104 for MSI/MSI-X
bnx2x 0000:04:00.1: irq 105 for MSI/MSI-X
bnx2x 0000:04:00.1: irq 106 for MSI/MSI-X
bnx2x 0000:04:00.1: irq 107 for MSI/MSI-X
bnx2x 0000:04:00.1: irq 108 for MSI/MSI-X
bnx2x: eth5: using MSI-X  IRQs: sp 103  fp[0] 105 ...
[ message continues ]

Thank you for this debug information! You saved one email round trip :)

However, I still need some more information about the FW version and
nvram settings. Can you please send me the output of ethtool -i and
ethtool -e? Since ethtool -e is quite long, it is best to send it as an
I assume it is the same when you unplug the module and insert it again
when the driver is already up and running, right? I also assumed you
verified it is firmly plugged in (no offense, bug I had to ask...)

Eilon


--

[ message continues ]

Almost everything seems to be in order. Almost - since you don't get
link... I don't think I have tried using this kind of Direct Attach
Cable - so maybe it just needs some more time. Let's see if the
following makes any difference (other than delay the failure for another
2.7 seconds):

diff --git a/drivers/net/bnx2x_link.c b/drivers/net/bnx2x_link.c
index ff70be8..bcee38c 100644
--- a/drivers/net/bnx2x_link.c
+++ b/drivers/net/bnx2x_link.c
@@ -3113,7 +3113,7 @@ static u8
bnx2x_wait_for_sfp_module_initialized(struct link_params *params)
        u16 timeout;
        /* Initialization time after hot-plug may take up to 300ms for
some
        phys type ( e.g. JDSU ) */
-       for (timeout = 0; timeout < 60; timeout++) {
+       for (timeout = 0; timeout < 600; timeout++) {
                if (bnx2x_read_sfp_module_eeprom(params, 1, 1, &val)
                    == 0) {
                        DP(NETIF_MSG_LINK, "SFP+ module initialization "

If it does help, be sure to let me know how much time it took (you
should have this debug print).

Regards,
Eilon


--

[ message continues ]

Still no luck. :( The kernel printed many more "Got bad status 0x0 when 
reading from SFP+ EEPROM" messages. Finally I got:
  "SFP+ module is not initialized".

Best regards,

			Krzysztof Olędzki
--

[ message continues ]

Some simple/simplistic thoughts/questions...

Has the DAC been used successfully prior to this?

Do you have another HP ProCurve 10-GbE SFP+ 7m Direct Attach Cable (J9285B) to try?

There's a transceiver and presumably an EEPROM at both ends of a DAC right?  If 
the EEPROM at one end were "bad" might the 57711 be happier with the other end 
of the DAC?  Getting some sort of error message at the switch side, which may 
(or may not) have more detailed diagnostics might help.

rick jones
--

[ message continues ]

Yes. It was successfully used to connect two HP switches, before I 
received SFP+ SR modules, that allowed me to put the switches into 




There is no error message at the switch side. The switch shows that 
everything is correct.

Best regards,

				Krzysztof Olędzki
--

[ message continues ]

Krzysztof,

I would like to check in what speed the I2C is running at, and while we
are there, to make sure it is at low speed (100KHz and not 400KHz). Can
you please add this patch and let me know what you see:

diff --git a/drivers/net/bnx2x_link.c b/drivers/net/bnx2x_link.c
index ff70be8..6e05f99 100644
--- a/drivers/net/bnx2x_link.c
+++ b/drivers/net/bnx2x_link.c
@@ -2766,6 +2766,21 @@ static u8
bnx2x_8727_read_sfp_module_eeprom(struct link_params *params,
                      MDIO_PMA_REG_SFP_TWO_WIRE_CTRL,
                      &val);
 
+       /* Make sure we are working in 100KHz */
+       bnx2x_cl45_read(bp, port,
+                     PORT_HW_CFG_XGXS_EXT_PHY_TYPE_BCM8727,
+                     ext_phy_addr,
+                     MDIO_PMA_DEVAD,
+                     8005,
+                     &val);
+       BNX2X_ERR("I2C 2W speed 0x%x, bit 8 %d\n", val, val & 8);
+       bnx2x_cl45_write(bp, port,
+                      ext_phy_type,
+                      ext_phy_addr,
+                      MDIO_PMA_DEVAD,
+                      8005,
+                      val & ~8);
+
        /* Set the read command byte count */
        bnx2x_cl45_write(bp, port,
                       ext_phy_type,

Since we are already off for the weekend, I couldn't find a machine with
this kind of board to test it for myself. On top of that, I don't have
this type of DAC cable, so I need your help in this debug process.

Thanks,
Eilon



--

[ message continues ]

<CUT>

[bnx2x_8727_read_sfp_module_eeprom:2774(eth4)]I2C 2W speed 0x0, bit 8 0

No problem. There is no need to ask for a help a person who depends on 
your support. ;) I would like to solve it as soon as possible so I'm 
willing to test everything you send me. ;)

best regards,

			Krzysztof Olędzki

--

[ message continues ]

Hello again,

To make sure the NICs are OK I made a test and plugged a SFP+ SR module (HP J9150A):

[bnx2x_8727_handle_mod_abs:4639(eth4)]MOD_ABS indication show module is present
[bnx2x_8727_read_sfp_module_eeprom:2774(eth4)]I2C 2W speed 0x0, bit 8 0
[bnx2x_8727_read_sfp_module_eeprom:2835(eth4)]Got bad status 0x0 when reading from SFP+ EEPROM
[bnx2x_8727_read_sfp_module_eeprom:2774(eth4)]I2C 2W speed 0x0, bit 8 0
[bnx2x_8727_read_sfp_module_eeprom:2835(eth4)]Got bad status 0x0 when reading from SFP+ EEPROM
[bnx2x_8727_read_sfp_module_eeprom:2774(eth4)]I2C 2W speed 0x0, bit 8 0
[bnx2x_8727_read_sfp_module_eeprom:2835(eth4)]Got bad status 0x0 when reading from SFP+ EEPROM
[bnx2x_8727_read_sfp_module_eeprom:2774(eth4)]I2C 2W speed 0x0, bit 8 0
[bnx2x_8727_read_sfp_module_eeprom:2835(eth4)]Got bad status 0x0 when reading from SFP+ EEPROM
[bnx2x_8727_read_sfp_module_eeprom:2774(eth4)]I2C 2W speed 0x0, bit 8 0
[bnx2x_8727_read_sfp_module_eeprom:2835(eth4)]Got bad status 0x0 when reading from SFP+ EEPROM
[bnx2x_8727_read_sfp_module_eeprom:2774(eth4)]I2C 2W speed 0x0, bit 8 0
[bnx2x_wait_for_sfp_module_initialized:3138(eth4)]SFP+ module initialization took 25 ms
[bnx2x_sfp_module_detection:3197(eth4)]SFP+ module plugged in/out detected on port 0
[bnx2x_8727_read_sfp_module_eeprom:2774(eth4)]I2C 2W speed 0x0, bit 8 0
[bnx2x_get_edc_mode:2930(eth4)]Optic module detected
[bnx2x_8727_read_sfp_module_eeprom:2774(eth4)]I2C 2W speed 0x0, bit 8 0
[bnx2x_get_edc_mode:2954(eth4)]EDC mode is set to 0x44
[bnx2x_8727_read_sfp_module_eeprom:2774(eth4)]I2C 2W speed 0x0, bit 8 0
[bnx2x_8727_read_sfp_module_eeprom:2774(eth4)]I2C 2W speed 0x0, bit 8 0
bnx2x: Warning: Unqualified SFP+ module detected on eth4, Port 0 from MergeOptics GmbH part number TRX10GVP2010CA01
[bnx2x_sfp_module_detection:3205(eth4)]Module verification failed!!

So, both NICs and SFP+ DA cables are OK. The only problem we have is to make them talk together. :|

Best regards,

			Krzysztof Olędzki
--

[ message continues ]