Cc: Jason Gunthorpe <jgunthorpe@...>, Rik van Riel <riel@...>, Andrea Arcangeli <andrea@...>, <a.p.zijlstra@...>, <izike@...>, Roland Dreier <rdreier@...>, <steiner@...>, <linux-kernel@...>, <avi@...>, <linux-mm@...>, <daniel.blueman@...>, Robin Holt <holt@...>, <general@...>, Andrew Morton <akpm@...>, <kvm-devel@...>
HPC environments won't be amenable to a pessimistic approach of
synchronizing before every data transfer. RDMA is assumed to be a
low-level data movement mechanism that has no implied
synchronization. In some parallel programming models, it's not
uncommon to use RDMA to send 8-byte messages. It can be difficult to
make and hold guarantees about in-memory pages when many concurrent
RDMA operations are in flight (not uncommon in reasonably large
machines). Some of the in-memory page information could be shared
with some form of remote caching strategy but then it's a different
problem with its own scalability challenges.
I think there are very potential clients of the interface when an
optimistic approach is used. Part of the trick, however, has to do
with being able to re-start transfers instead of buffering the data
or making guarantees about delivery that could cause deadlock (as was
alluded to earlier in this thread). InfiniBand is constrained in
this regard since it requires message-ordering between endpoints (or
queue pairs). One could argue that this is still possible with IB,
at the cost of throwing more packets away when a referenced page is
not in memory. With this approach, the worse case demand paging
scenario is met when the active working set of referenced pages is
larger than the amount physical memory -- but HPC applications are
already bound by this anyway.
You'll find that Quadrics has the most experience in this area and
that their entire architecture is adapted to being optimistic about
demand paging in RDMA transfers -- they've been maintaining a patchset
to do this for years.
. . christian
--
