>
>Thanks,
>
>Céline Bourde.
>
>
>Tom Talpey wrote:
>> At 06:56 AM 4/27/2009, Celine Bourde wrote:
>>
>>> Thanks for the explanation.
>>> Let me know if you have additional information.
>>>
>>> We have a contact at Mellanox. I will contact him.
>>>
>>> Thanks,
>>>
>>> Céline.
>>>
>>> Vu Pham wrote:
>>>
>>>> Celine,
>>>>
>>>> I'm seeing mlx4 in the log so it is connectX.
>>>>
>>>> nfsrdma does not work with any official connectX' fw release 2.6.0
>>>> because of fast registering work request problems between nfsrdma and
>>>> the firmware.
>>>>
>>
>> There is a very simple workaround if you don't have the latest mlx4 firmware.
>>
>> Just set the client to use the all-physical memory registration
>mode. This will
>> avoid making unsupported reregistration requests, which the firmware
>advertised.
>>
>> Before mounting, enter (as root)
>>
>> sysctl -w sunrpc.rdma_memreg_strategy = 6
>>
>> The client should work properly after this.
>>
>> If you do have access to the fixed firmware, I recommend using the default
>> setting (5) as it provides greater safety on the client.
>>
>> Tom.
>>
>>
>>>> We are currently debugging/fixing those problems.
>>>>
>>>> Do you have direct contact with Mellanox field application engineer?
>>>> Please contact him/her.
>>>> If not I can send you a contact on private channel.
>>>>
>>>> thanks,
>>>> -vu
>>>>
>>>>
>>>>> Hi Celine,
>>>>>
>>>>> What HCA do you have on your system? Is it ConnectX? If yes, what is
>>>>> its firmware version?
>>>>>
>>>>> -vu
>>>>>
>>>>>
>>>>>> Hey Celine,
>>>>>>
>>>>>> Thanks for gathering all this info! So the rdma connections work
>>>>>> fine with everything _but_ nfsrdma. And errno 103 indicates the
>>>>>> connection was aborted, maybe by the server (since no failures are
>>>>>> logged by the client).
>>>>>>
>>>>>>
>>>>>> More below:
>>>>>>
>>>>>>
>>>>>> Celine Bourde wrote:
>>>>>>
>>>>>>> Hi Steve,
>>>>>>>
>>>>>>> This email summarizes the situation:
>>>>>>>
>>>>>>> Standard mount -> OK
>>>>>>> ---------------------
>>>>>>>
>>>>>>> [root@twind ~]# mount -o rw 192.168.0.215:/vol0 /mnt/
>>>>>>> Command works fine.
>>>>>>>
>>>>>>> rdma mount -> KO
>>>>>>> -----------------
>>>>>>>
>>>>>>> [root@twind ~]# mount -o rdma,port=2050 192.168.0.215:/vol0 /mnt/
>>>>>>> Command blocks ! I should perform Ctr+C to kill process.
>>>>>>>
>>>>>>> or
>>>>>>>
>>>>>>> [root@twind ofa_kernel-1.4.1]# strace mount.nfs 192.168.0.215:/vol0
>>>>>>> /mnt/ -o rdma,port=2050
>>>>>>> [..]
>>>>>>> fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
>>>>>>> connect(3, {sa_family=AF_INET, sin_port=htons(610),
>>>>>>> sin_addr=inet_addr("127.0.0.1")}, 16) = 0
>>>>>>> fcntl(3, F_SETFL, O_RDWR) = 0
>>>>>>> sendto(3,
>>>>>>>
>>>>>>>
>>>
>"-357__PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2_60__PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2_"...,
>>>
>>>>>>> 40, 0, {sa_family=AF_INET, sin_port=htons(610),
>>>>>>> sin_addr=inet_addr("127.0.0.1")}, 16) = 40
>>>>>>> poll([{fd=3, events=POLLIN}], 1, 3000) = 1 ([{fd=3, revents=POLLIN}])
>>>>>>> recvfrom(3, "-357__PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2___PLACEHOLDER__2_",
>>>>>>> 8800, MSG_DONTWAIT, {sa_family=AF_INET, sin_port=htons(610),
>>>>>>> sin_addr=inet_addr("127.0.0.1")}, [16]) = 24
>>>>>>> close(3) = 0
>>>>>>> mount("192.168.0.215:/vol0", "/mnt", "nfs", 0,
>>>>>>> "rdma,port=2050,addr=192.168.0.215"
>>>>>>> ..same problem
>>>>>>>
>>>>>>> [root@twind tmp]# dmesg
>>>>>>> rpcrdma: connection to 192.168.0.215:2050 on mlx4_0, memreg 5 slots
>>>>>>> 32 ird 16
>>>>>>> rpcrdma: connection to 192.168.0.215:2050 closed (-103)
>>>>>>> rpcrdma: connection to 192.168.0.215:2050 on mlx4_0, memreg 5 slots
>>>>>>> 32 ird 16
>>>>>>> rpcrdma: connection to 192.168.0.215:2050 closed (-103)
>>>>>>> rpcrdma: connection to 192.168.0.215:2050 on mlx4_0, memreg 5 slots
>>>>>>> 32 ird 16
>>>>>>> rpcrdma: connection to 192.168.0.215:2050 closed (-103)
>>>>>>> rpcrdma: connection to 192.168.0.215:2050 on mlx4_0, memreg 5 slots
>>>>>>> 32 ird 16
>>>>>>> rpcrdma: connection to 192.168.0.215:2050 closed (-103)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Is there anything logged on the server side?
>>>>>>
>>>>>> Also, can you try this again, but on both systems do this before
>>>>>> attempting the mount:
>>>>>>
>>>>>> echo 32768 > /proc/sys/sunrpc/rpc_debug
>>>>>>
>>>>>> This will enable all the rpc trace points and add a bunch of logging
>>>>>> to /var/log/messages.
>>>>>> Maybe that will show us something. It think the server is aborting
>>>>>> the connection for some reason.
>>>>>>
>>>>>> Steve.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> general mailing list
>>>>>>
general@lists.openfabrics.org
>>>>>>
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>>>
>>>>>> To unsubscribe, please visit
>>>>>>
http://openib.org/mailman/listinfo/openib-general
>>>>>>
>>>>> _______________________________________________
>>>>> general mailing list
>>>>>
general@lists.openfabrics.org
>>>>>
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>>
>>>>> To unsubscribe, please visit
>>>>>
http://openib.org/mailman/listinfo/openib-general
>>>>>
>>>> _______________________________________________
>>>> general mailing list
>>>>
general@lists.openfabrics.org
>>>>
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>
>>>> To unsubscribe, please visit
>>>>
http://openib.org/mailman/listinfo/openib-general
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> general mailing list
>>>
general@lists.openfabrics.org
>>>
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>
>>> To unsubscribe, please visit
>http://openib.org/mailman/listinfo/openib-general
>>>
>>>
>>
>>
>>
>>
>
>
