login
Search
As you probably know there is a trend in enterprise computing towards networked storage. This is illustrated by the emergence during the past few years of standards like SRP (SCSI RDMA Protocol), iSCSI (Internet SCSI) and iSER (iSCSI Extensions for RDMA). Two different pieces of software are necessary to make networked storage possible: initiator software and target software. As far as I know there exist three different SCSI target implementations for Linux: - The iSCSI Enterprise Target Daemon (IETD, http://iscsitarget.sourceforge.net/); - The Linux SCSI Target Framework (STGT, http://stgt.berlios.de/); - The Generic SCSI Target Middle Level for Linux project (SCST, http://scst.sourceforge.net/). Since I was wondering which SCSI target software would be best suited for an InfiniBand network, I started evaluating the STGT and SCST SCSI target implementations. Apparently the performance difference between STGT and SCST is small on 100 Mbit/s and 1 Gbit/s Ethernet networks, but the SCST target software outperforms the STGT software on an InfiniBand network. See also the following thread for the details: http://sourceforge.net/mailarchive/forum.php?thread_name=e2e108260801170127w2937b2afg9.... About the design of the SCST software: while one of the goals of the STGT project was to keep the in-kernel code minimal, the SCST project implements the whole SCSI target in kernel space. SCST is implemented as a set of new kernel modules, only minimal changes to the existing kernel are necessary before the SCST kernel modules can be used. This is the same approach that will be followed in the very near future in the OpenSolaris kernel (see also http://opensolaris.org/os/project/comstar/). More information about the design of SCST can be found here: http://scst.sourceforge.net/doc/scst_pg.html. My impression is that both the STGT and SCST projects are well designed, well maintained and have a considerable user base. According to the SCST ...
That doesn't seem to pull up a thread. However, I assume it's these figures: ............................................................................................. . . STGT read SCST read . STGT read SCST read . . . performance performance . performance performance . . . (0.5K, MB/s) (0.5K, MB/s) . (1 MB, MB/s) (1 MB, MB/s) . ............................................................................................. . Ethernet (1 Gb/s network) . 77 78 . 77 89 . . IPoIB (8 Gb/s network) . 163 185 . 201 239 . . iSER (8 Gb/s network) . 250 N/A . 360 N/A . . SRP (8 Gb/s network) . N/A 421 . N/A 683 . ............................................................................................. On the comparable figures, which only seem to be IPoIB they're showing a The two target architectures perform essentially identical functions, so there's only really room for one in the kernel. Right at the moment, it's STGT. Problems in STGT come from the user<->kernel boundary which can be mitigated in a variety of ways. The fact that the figures are pretty much comparable on non IB networks shows this. I really need a whole lot more evidence than at worst a 20% performance difference on IB to pull one implementation out and replace it with another. Particularly as there's no real evidence that STGT can't be tweaked to recover the 20% even on IB. James --
> . . STGT read SCST read . STGT read SCST read . > . . performance performance . performance performance . > . . (0.5K, MB/s) (0.5K, MB/s) . (1 MB, MB/s) (1 MB, MB/s) . > . iSER (8 Gb/s network) . 250 N/A . 360 N/A . > . SRP (8 Gb/s network) . N/A 421 . N/A 683 . > On the comparable figures, which only seem to be IPoIB they're showing a > 13-18% variance, aren't they? Which isn't an incredible difference. Maybe I'm all wet, but I think iSER vs. SRP should be roughly comparable. The exact formatting of various messages etc. is different but the data path using RDMA is pretty much identical. So the big difference between STGT iSER and SCST SRP hints at some big difference in the efficiency of the two implementations. - R. --
On Tue, 29 Jan 2008 13:31:52 -0800
iSER has parameters to limit the maximum size of RDMA (it needs to
repeat RDMA with a poor configuration)?
Anyway, here's the results from Robin Humble:
iSER to 7G ramfs, x86_64, centos4.6, 2.6.22 kernels, git tgtd,
initiator end booted with mem=512M, target with 8G ram
direct i/o dd
write/read 800/751 MB/s
dd if=/dev/zero of=/dev/sdc bs=1M count=5000 oflag=direct
dd of=/dev/null if=/dev/sdc bs=1M count=5000 iflag=direct
http://www.mail-archive.com/linux-scsi@vger.kernel.org/msg13502.html
I think that STGT is pretty fast with the fast backing storage.
I don't think that there is the notable perfornace difference between
kernel-space and user-space SRP (or ISER) implementations about moving
data between hosts. IB is expected to enable user-space applications
to move data between hosts quickly (if not, what can IB provide us?).
I think that the question is how fast user-space applications can do
I/Os ccompared with I/Os in kernel space. STGT is eager for the advent
of good asynchronous I/O and event notification interfances.
One more possible optimization for STGT is zero-copy data
transfer. STGT uses pre-registered buffers and move data between page
cache and thsse buffers, and then does RDMA transfer. If we implement
own caching mechanism to use pre-registered buffers directly with (AIO
and O_DIRECT), then STGT can move data without data copies.
--
Both Robin (iser/stgt) and Bart (scst/srp) using ramfs
Robin's numbers come from DDR IB HCAs
Bart's numbers come from SDR IB HCAs:
Results with /dev/ram0 configured as backing store on the
target (buffered I/O):
Read Write Read
Write
performance performance
performance performance
(0.5K, MB/s) (0.5K, MB/s) (1 MB,
MB/s) (1 MB, MB/s)
STGT + iSER 250 48 349
781
SCST + SRP 411 66 659
746
Results with /dev/ram0 configured as backing store on the
target (direct I/O):
Read Write Read
Write
performance performance
performance performance
(0.5K, MB/s) (0.5K, MB/s) (1 MB,
MB/s) (1 MB, MB/s)
STGT + iSER 7.9 9.8 589
647
SCST + SRP 12.3 9.7 811
794
http://www.mail-archive.com/linux-scsi@vger.kernel.org/msg13514.html
Here are my numbers with DDR IB HCAs, SCST/SRP 5G /dev/ram0
block_io mode, RHEL5 2.6.18-8.el5
direct i/o dd
write/read 1100/895 MB/s
dd if=/dev/zero of=/dev/sdc bs=1M count=5000 oflag=direct
dd of=/dev/null if=/dev/sdc bs=1M count=5000 iflag=direct
buffered i/o dd
write/read 950/770 MB/s
dd if=/dev/zero of=/dev/sdc bs=1M count=5000
dd of=/dev/null if=/dev/sdc bs=1M count=5000
So when using DDR IB hcas:
stgt/iser scst/srp
direct I/O 800/751 1100/895
buffered I/O 1109/350 950/770
--
Please specify which parameters you are referring to. As you know I had already repeated my tests with ridiculously high values for the following iSER parameters: FirstBurstLength, MaxBurstLength and MaxRecvDataSegmentLength (16 MB, which is more than the 1 MB block size specified to dd). Bart Van Assche. --
On Wed, 30 Jan 2008 09:38:04 +0100 Sorry, I can't say. I don't know much about iSER. But seems that Pete and Robin can get the better I/O performance - line speed ratio with STGT. The version of OpenIB might matters too. For example, Pete said that STGT reads loses about 100 MB/s for some transfer sizes for some transfer sizes due to the OpenIB version difference or other unclear reasons. http://article.gmane.org/gmane.linux.iscsi.tgt.devel/135 It's fair to say that it takes long time and need lots of knowledge to get the maximum performance of SAN, I think. I think that it would be easier to convince James with the detailed analysis (e.g. where does it take so long, like Pete did), not just 'dd' performance results. Pushing iSCSI target code into mainline failed four times: IET, SCST, STGT (doing I/Os in kernel in the past), and PyX's one (*1). iSCSI target code is huge. You said SCST comprises 14,000 lines, but it's not iSCSI target code. The SCSI engine code comprises 14,000 lines. You need another 10,000 lines for the iSCSI driver. Note that SCST's iSCSI driver provides only basic iSCSI features. PyX's iSCSI target code implemenents more iSCSI features (like MC/S, ERL2, etc) and comprises about 60,000 lines and it still lacks some features like iSER, bidi, etc. I think that it's reasonable to say that we need more than 'dd' results before pushing about possible more than 60,000 lines to mainline. (*1) http://linux-iscsi.org/ --
Tomo, please stop counting in-kernel lines only (see http://lkml.org/lkml/2007/4/24/364). The amount of the overall project --
Robin Humble was using a DDR InfiniBand network, while my tests were performed with an SDR InfiniBand network. Robin's results can't be directly compared to my results. Pete Wyckoff's results (http://www.osc.edu/~pw/papers/wyckoff-iser-snapi07-talk.pdf) are hard to interpret. I have asked Pete which of the numbers in his test can Pete wrote about a degradation from 600 MB/s to 500 MB/s for reads with STGT+iSER. In my tests I measured 589 MB/s for reads (direct I/O), which matches with the better result obtained by Pete. Note: the InfiniBand kernel modules I used were those from the 2.6.22.9 kernel, not from the OFED distribution. Bart. --
On Wed, 30 Jan 2008 14:10:47 +0100 I know that you use different hardware. I used 'ratio' word. BTW, you said the performance difference of dio READ is 38% but I I don't know he used the same benchmark software so I don't think that we can compare them. All I tried to say is the OFED version might has big effect on the I'm talking about a target machine (I think that Pete was also talking about OFED on his target machine). STGT uses OFED libraries, I think. --
Let's start with summarizing the relevant numbers from Robin's measurements and my own measurements. Maximum bandwidth of the underlying physical medium: 2000 MB/s for a DDR 4x InfiniBand network and 1000 MB/s for a SDR 4x InfiniBand network. Maximum bandwidth reported by the OFED ib_write_bw test program: 1473 MB/s for Robin's setup and 933 MB/s for my setup. These numbers match published ib_write_bw results (see e.g. figure 11 in http://www.usenix.org/events/usenix06/tech/full_papers/liu/liu_html/index.html or chapter 7 in http://www.voltaire.com/ftp/rocks/HCA-4X0_Linux_GridStack_4.3_Release_Notes_DOC-00171-...) Throughput measured for communication via STGT + iSER to a remote RAM disk via direct I/O with dd: 800 MB/s for writing and 751 MB/s for reading in Robin's setup, and 647 MB/s for writing and 589 MB/s for reading in my setup. From this we can compute the I/O-performance to ib_write_bw bandwidth: 54 % for writing and 51 % for reading in Robin's setup, and 69 % for writing and 63 % for reading in my setup. Or a slightly better utilization of the bandwidth in my setup than in Robin's setup. This is no surprise -- the faster a communication link is, the harder it is to use all of the available bandwidth. So why did you state that in Robin's tests the I/O performance to line speed ratio was better than in my tests ? Bart Van Assche. --
Greetings all, The PyX storage engine supports a scatterlist linked list algorithm that maps any sector count + sector size combination down to contiguous struct scatterlist arrays across (potentially) multiple Linux storage subsystems from a single CDB received on a initiator port. This design was a consequence of a requirement for running said engine on Linux v2.2 and v2.4 across non cache coherent systems (MIPS R5900-EE) using a single contiguous memory block mapped into struct buffer_head for PATA access, and struct scsi_cmnd access on USB storage. Note that this was before struct bio and struct scsi_request existed.. The PyX storage engine as it exists at Linux-iSCSI.org today can be thought of as a hybrid OSD processing engine, as it maps storage object memory across a number of tasks from a received command CDB. The ability to pass in pre allocated memory from an RDMA capable adapter, as well as allocated internally (ie: traditional iSCSI without open_iscsi's struct skbuff rx zero-copy) is inherient in the design of the storage engine. The lacking Bidi support can be attributed to lack of greater support (and hence user interest) in Bidi, but I am really glad to see this getting into the SCSI ML and STGT, and is certainly of interest in see in Linux as well. This is pretty easy to add in iSCSI with an AHS The 60k lines of code also includes functionality (the SE mirroring comes to mind) that I do not plan to push towards mainline, along with other legacy bits so we can build on earlier v2.6 embedded platforms. The existing Target mode LIO-SE that provides linked list scatterlist mapping algorithm that is similar to what Jens and Rusty have been working on, and is under 14k lines including the switch(cdb[0]) + function pointer assignment to per CDB specific structure that is called potentially out-of-order in the RX side context of the CmdSN state machine in RFC-3720. The current SE is also lacking the very SCSI specific task management state machines that not ...
I can run disktest on the same setups I ran dd on. This will take some time however. Disktest is new to me -- any hints with regard to suitable combinations of command line parameters are welcome. The most recent version I could find on http://ltp.sourceforge.net/ is ltp-20071231. Bart Van Assche. --
Hi Bart, I posted some numbers with traditional iSCSI on Neterion Xframe I 10 Gb/sec with LRO back in 2005 with disktest on the 1st generation x86_64 hardware available at the time. These tests where designed to show the performance advantages of internexus multiplexing that is available within traditional iSCSI, as well as iSER. The disktest parameters that I used are listed in the following thread: https://www.redhat.com/archives/dm-devel/2005-April/msg00013.html --
Disktest was already referenced in the beginning of the performance comparison thread, but its results are not very interesting if we are going to find out, which implementation is more effective, because in the modes, in which usually people run this utility, it produces latency insensitive workload (multiple threads working in parallel). So, such multithreaded disktests results will be different between STGT and SCST only if STGT's implementation will get target CPU bound. If CPU on the target is powerful enough, even extra busy loops in the STGT or SCST hot path code will change nothing. Additionally, multithreaded disktest over RAM disk is a good example of a synthetic benchmark, which has almost no relation with real life workloads. But people like it, because it produces nice looking results. Actually, I don't know what kind of conclusions it is possible to make from disktest's results (maybe only how throughput gets bigger or slower with increasing number of threads?), it's a good stress test tool, but --
There are other issues with disktest, in that you can easily specify option combinations that generate apparently 5+ GB/s of IO, though actual traffic over the link to storage is very low. Caveat disktest I agree. The backing store should be a disk for it to have meaning, Unfortunately, I agree. Bonnie++, dd tests, and a few others seem to bear far closer to "real world" tests than disktest and iozone, the latter of which does more to test the speed of RAM cache and system call Here is what I have run: disktest -K 8 -B 256k -I F -N 20000000 -P A -w /big/file disktest -K 8 -B 64k -I F -N 20000000 -P A -w /big/file disktest -K 8 -B 1k -I B -N 2000000 -P A /dev/sdb2 and many others. Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 --
I have ran some tests with Bonnie++, but found out that on a fast network like IB the filesystem used for the test has a really big impact on the test results. If anyone has a suggestion for a better test than dd to compare the performance of SCSI storage protocols, please let it know. Bart Van Assche. --
This is true of the file systems when physically directly connected to the unit as well. Some file systems are designed with high performance Hmmm... if you care about the protocol side, I can't help. Our users are more concerned with the file system side, so this is where we focus Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 --
xdd on /dev/sda, sdb, etc. using -dio to do direct IO seems to work decently, though it is hard (ie, impossible) to get a repeatable sequence of IO when using higher queue depths, as it uses threads to generate multiple requests. You may also look at sgpdd_survey from Lustre's iokit, but I've not done much with that -- it uses the sg devices to send lowlevel SCSI commands. I've been playing around with some benchmark code using libaio, but it's not in generally usable shape. xdd: http://www.ioperformance.com/products.htm Lustre IO Kit: http://manual.lustre.org/manual/LustreManual16_HTML/DynamicHTML-20-1.html -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office --
This utility seems to be a good one, but it's basically the same as Yes, it might be worth to try. Since fundamentally it's the same as O_DIRECT dd, but with a bit less overhead on the initiator side (hence less initiator side latency), most likely it will show ever bigger --
I would suggest you to try something from real life, like: - Copying large file tree over a single or multiple IB links - Measure of some DB engine's TPC --
Forgot to mention. During those tests make sure that imported devices from both SCST and STGT report in the kernel log the same write cache and FUA capabilities, since they significantly affect initiator's behavior. Like: sd 4:0:0:5: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA --
I think the really interesting numbers are the difference for bulk I/O between kernel and userspace on both traditional iSCSI and the RDMA enabled flavours. I have not been able to determine anything earth shattering from the current run of kernel vs. userspace tests, nor which method of implementation for iSER, SRP, and generic Storage Engine are 'more effective' for that case. Performance and latency to real storage would make alot more sense for the kernel vs. user case. Also workloads against software LVM and Linux MD block devices would be of interest as these would be some of the more typical deployments that would be in the field, and is what Linux-iSCSI.org uses for our production cluster storage today. Having implemented my own iSCSI and SCSI Target mode Storage Engine leads me to believe that putting logic in userspace is probably a good idea in the longterm. If this means putting the entire data IO path into userspace for Linux/iSCSI, then there needs to be a good reason why this will not not scale to multi-port 10 Gb/sec engines in traditional and RDMA mode if we need to take this codepath back into the kernel. The end goal is to have the most polished and complete storage engine and iSCSI stacks designs go upstream, which is something I think we can all agree on. Also, with STGT being a pretty new design which has not undergone alot of optimization, perhaps profiling both pieces of code against similar tests would give us a better idea of where userspace bottlenecks reside. Also, the overhead involved with traditional iSCSI for bulk IO from kernel / userspace would also be a key concern for a much larger set of users, as iSER and SRP on IB is a pretty small userbase and will Yes, people like to claim their stacks are the fastest with RAM disk benchmarks. But hooking up their fast network silicon to existing storage hardware and OS storage subsystems and software is where the Being able to have a best case baseline with disktest for kernel vs. user ...
Two important trends in data center technology are server consolidation and storage consolidation. A.o. every web hosting company can profit from a fast storage solution. I wouldn't call this a small user base. Regarding iSER and SRP on IB: InfiniBand is today the most economic solution for a fast storage network. I do not know which technology will be the most popular for storage consolidation within a few years -- this can be SRP, iSER, IPoIB + SDP, FCoE (Fibre Channel over Ethernet) or maybe yet another technology. No matter which technology becomes the most popular for storage applications, there will be a need for high-performance storage software. References: * Michael Feldman, Battle of the Network Fabrics, HPCwire, December 2006, http://www.hpcwire.com/hpc/1145060.html * NetApp, Reducing Data Center Power Consumption Through Efficient Storage, February 2007, http://www.netapp.com/ftp/wp-reducing-datacenter-power-consumption.pdf Bart Van Assche. --
I meant small referring to storage on IB fabrics which has usually been in the research and national lab settings, with some other vendors offering IB as an alternative storage fabric for those who [w,c]ould not wait for 10 Gb/sec copper Ethernet and Direct Data Placement to come online. These types of numbers compared to say traditional iSCSI, that is getting used all over the place these days in areas I won't bother listing here. As for the future, I am obviously cheering for IP storage fabrics, in particular 10 Gb/sec Ethernet and Direct Data Placement in concert with iSCSI Extentions for RDMA to give the data center a high performance, low latency transport that can do OS independent storage multiplexing and recovery across multiple independently developed implementions. Also avoiding lock-in from un-interoptable storage transports (espically on the high end) that had plauged so many vendors in years past has become an real option in the past few years with a IETF defined block level storage protocol. We are actually going on four years since RFC-3720 was ratified. (April 2004) Making the 'enterprise' ethernet switching equipment go from millisecond to nanosecond latency in a whole different story that goes beyond my area of expertise. I know there is one startup (Fulcrum Micro) who is working on this problem and seems to be making some good progress. --
InfiniBand has several advantages over 10 Gbit/s Ethernet (the list below probably isn't complete): - Lower latency. Communication latency is not only determined by the latency of a switch. The whole InfiniBand protocol stack was designed with low latency in mind. Low latency is really important for database software that accesses storage over a network. - High-availability is implemented at the network layer. Suppose that a group of servers has dual-port network interfaces and is interconnected via a so-called dual star topology, With an InfiniBand network, failover in case of a single failure (link or switch) is handled without any operating system or application intervention. With Ethernet, failover in case of a single failure must be handled either by the operating system or by the application. - You do not have to use iSER or SRP to use the bandwidth of an InfiniBand network effectively. The SDP (Sockets Direct Protocol) makes it possible that applications benefit from RDMA by using the very classic IPv4 Berkeley sockets interface. An SDP implementation in software is already available today via OFED. iperf reports 470 MB/s on single-threaded tests and 975 MB/s for a performance test with two threads on an SDR 4x InfiniBand network. These tests were performed with the OFED 1.2.5.4 SDP implementation. It is possible that future SDP implementations will perform even better. (Note: I could not yet get iSCSI over SDP working.) We should leave the choice of networking technology open -- both Ethernet and InfiniBand have specific advantages. See also: InfiniBand Trade Association, InfiniBand Architecture Specification Release 1.2.1, http://www.infinibandta.org/specs/register/publicspec/ Bart Van Assche. --
Which parts of the PyX source code are licensed under the GPL and which parts are closed source ? A Google query for PyX + iSCSI showed information about licensing deals. Licensing deals can only be closed for software that is not entirely licensed under the GPL. Bart Van Assche. --
I was using the name PyX to give an historical context to the discussion. :-) In more recent times, I have been using the name "LIO Target Stack" and "LIO Storage Engine" to refer to Traditional RFC-3720 Target statemachines, and SCSI Processing engine implementation respectively. The codebase has matured significantly from the original codebase, as the Linux SCSI, ATA and Block subsystems envolved from v2.2, v2.4, v2.5 and modern v2.6, the LIO stack has grown (and sometimes shrunk) along with the following requirement; To support all possible storage devices on all subsystems on any hardware platform that Linux could be made to boot. Interopt with other non Linux SCSI subsystems was also an issue early in development.. If you can imagine a Solaris SCSI subsystem asking for T10 EVPD WWN information from a Linux/iSCSI Target with pre libata SATA drivers, you can probably guess just how time was spent looking at packet captures to figure out to make OS dependent (ie: outernexus) multipath to play nice. Note that PyX Target Code for Linux v2.6 has been available in source and binary form for a diverse array of Linux devices and environments since September 2007. Right around this time, the Linux-iSCSI.org Storage and Virtualization stack went online for the first time using OCFS2, PVM, HVM, LVM, RAID6 and of course, traditional RFC-3720 on 10 Gb/sec and 1 Gb/sec fabric. There have also been world's first storage research work and prototypes that have been developed with the LIO code. Information on these topics is available from the homepage, and a few links deep there are older projects and information about features inherent to the LIO Target and Storage Engine. One of my items for the v2.9 codebase in 2008 is start picking apart the current code and determining which pieces should be sent upstream for review. I have also been spending alot of time recently looking at the other available open source storage transport and processing stacks and seeing how Linux/iSCSI, ...
Regarding the PyX Target Code: I have found a link via which I can download a free 30-day demo. This means that a company is earning money via this target code and that the source code is not licensed under the GPL. This is fine, but it also means that today the PyX target code is not a candidate for inclusion in the Linux kernel, and that it is unlikely that all of the PyX target code (kernelspace + userspace) will be made available under GPL soon. Bart Van Assche. --
All of the kernel and C userspace code is open source and available from linux-iscsi.org and licensed under the GPL. There is the BSD licensed code from userspace (iSNS), as well as ISCSI and SCSI MIBs. As for what pieces of code will be going upstream (for kernel and/or userspace), LIO Target state machines and SE algoritims are definately some of the best examples of GPL code for production IP storage fabric and has gained maturity from people and resources applied to it in a number of respects. The LIO stack presents a number of possible options to get the diverse amount of hardware and software to work. Completely dismissing the available code is certainly a waste, and there are still significant amounts of functionality related to real-time administration, RFC-3720 MC/S and ERL=2 and generic SE functionality OS storage subsystems that only exist in LIO and our assoicated projects. A one obvious example is the LIO-VM project, which brings LIO active-active transport recovery and other Linux storage functionality to Vmware and Qemu images that can provide target mode IP storage fabric on x86 non-linux based hosts. A first of its kind in the Linux/iSCSI universe. Anyways, lets get back to the technical discussion. --
I found a statement on a web page that the ERL2 implementation is not included in the GPL version (http://zaal.org/iscsi/index.html). The above implies that this statement is incorrect. Tomo, are you the maintainer of this web page ? I'll try to measure the performance of the LIO Target Stack on the same setup on which I ran the other performance tests. Bart Van Assche. --
the 1Mb block size is a bit of a red herring. Unless you've specifically increased the max_sector_size and are using an sg_chain converted driver, on x86 the maximum possible transfer accumulation is 0.5MB. I certainly don't rule out that increasing the transfer size up from 0.5MB might be the way to improve STGT efficiency, since at an 1GB/s theoretical peak, that's roughly 2000 context switches per I/O; however, It doesn't look like you've done anything that will overcome the block layer limitations. James --
On Jan 30, 2008 5:34 PM, James Bottomley I did not publish the results, but I have also done tests with other block sizes. The other sizes I tested were between 0.1MB and 10MB. The performance difference for these other sizes compared to a block size of 1MB was small (smaller than the variance between individual tests results). Bart. --
The MRDSL parameter has no effect on iSER, as the RFC describes. How to transfer data to satisfy a command is solely up to the target. So you would need both big requests from the client, then look at how the target will send the data. I've only used 512 kB for the RDMA transfer size from the target, as it matches the default client size and was enough to get good performance out of my IB gear and minimizes resource consumption on the target. It's currently hard-coded as a #define. There is no provision in the protocol for the client to dictate the value. If others want to spend some effort trying to tune stgt for iSER, there are a fair number of comments in the code, including a big one that explains this RDMA transfer size issue. And I'll answer informed questions as I can. But I'm not particularly interested in arguing about which implementation is best, or trying to interpret bandwidth comparison numbers from poorly designed tests. It takes work to understand these issues. -- Pete --
Using such large values for FirstBurstLength will give you poor performance numbers for WRITE commands (with iSER). FirstBurstLength means how much data should you send as unsolicited data (i.e. without RDMA). It means that your WRITE commands were sent without RDMA. Erez --
Sorry, but I'm afraid you got this wrong. When the iSER transport is used instead of TCP, all data is sent via RDMA, including unsolicited data. If you have look at the iSER implementation in the Linux kernel (source files under drivers/infiniband/ulp/iser), you will see that all data is transferred via RDMA and not via TCP/IP. Bart Van Assche. --
Regardless of what the current implementation is, the behavior you (Bart) describe seems to disagree with http://www.ietf.org/rfc/rfc5046.txt. --
> Sorry, but I'm afraid you got this wrong. When the iSER transport is > used instead of TCP, all data is sent via RDMA, including unsolicited > data. If you have look at the iSER implementation in the Linux kernel > (source files under drivers/infiniband/ulp/iser), you will see that > all data is transferred via RDMA and not via TCP/IP. I think the confusion here is caused by a slight misuse of the term "RDMA". It is true that all data is always transported over an InfiniBand connection when iSER is used, but not all such transfers are one-sided RDMA operations; some data can be transferred using send/receive operations. - R. --
When you execute WRITE commands with iSCSI, it works like this: EDTL (Expected data length) - the data length of your command FirstBurstLength - the length of data that will be sent as unsolicited data (i.e. as immediate data with the SCSI command and as unsolicited data-out PDUs) If you use a high value for FirstBurstLength, all (or most) of your data will be sent as unsolicited data-out PDUs. These PDUs don't use the RDMA engine, so you miss the advantage of IB. If you use a lower value for FirstBurstLength, EDTL - FirstBurstLength bytes will be sent as solicited data-out PDUs. With iSER, solicited data-out PDUs are RDMA operations. I hope that I'm more clear now. Erez --
Hello Erez, Did you notice the e-mail Roland Dreier wrote on Februari 6, 2008 ? Or: data sent during the first burst is not transferred via one-sided remote memory reads or writes but via two-sided send/receive operations. At least on my setup, these operations are as fast as one-sided remote memory reads or writes. As an example, I obtained the following numbers on my setup (SDR 4x network); ib_write_bw: 933 MB/s. ib_read_bw: 905 MB/s. ib_send_bw: 931 MB/s. Bart Van Assche. --
According to these numbers one can think that you don't need RDMA at all, just send iSCSI PDUs over IB. The benchmarks that you use are synthetic IB benchmarks that are not equivalent to iSCSI over iSER. They just send IB packets. I'm not surprised that you got more or less the same performance because, AFAIK, ib_send_bw doesn't copy data (unlike iSCSI that has to copy data that is sent/received without RDMA). When you use RDMA with iSCSI (i.e. iSER), you don't need to create iSCSI PDUs and process them. The CPU is not busy as it is with iSCSI over TCP because no data copies are required. Another advantage is that you don't need header/data digest because the IB HW does that. Erez --
I agree that ib_write_bw / ib_read_bw / ib_send_bw performance results are not equivalent to iSCSI over iSER. The reason that I included these performance results was to illustrate that two-sided data As far as I know, when using iSER, the FirstBurstLength bytes of data are sent via two-sided data transfers, and there is no CPU intervention required to transfer the data itself over the IB network. Bart Van Assche. --
Great! So, you are going to duplicate Linux page cache in the user space. You will continue keeping the in-kernel code as small as possible and its mainteinership effort as low as possible by the cost that the user space part's code size and complexity (and, hence, its mainteinership effort) will rocket to the sky. Apparently, this doesn't --
On Jan 29, 2008 9:42 PM, James Bottomley Are you saying that users who need an efficient iSCSI implementation should switch to OpenSolaris ? The OpenSolaris COMSTAR project involves the migration of the existing OpenSolaris iSCSI target daemon from userspace to their kernel. The OpenSolaris developers are spending time on this because they expect a significant performance My measurements on a 1 GB/s InfiniBand network have shown that the current SCST implementation is able to read data via direct I/O at a rate of 811 GB/s (via SRP) and that the current STGT implementation is able to transfer data at a rate of 589 MB/s (via iSER). That's a performance difference of 38%. And even more important, the I/O latency of SCST is significantly lower than that of STGT. This is very important for database workloads -- the I/O pattern caused by database software is close to random I/O, and database software needs low latency I/O in order to run efficiently. In the thread with the title "Performance of SCST versus STGT" on the SCST-devel / STGT-devel mailing lists not only the raw performance numbers were discussed but also which further performance improvements are possible. It became clear that the SCST performance can be improved further by implementing a well known optimization (zero-copy I/O). Fujita Tomonori explained in the same thread that it is possible to improve the performance of STGT further, but that this would require a lot of effort (implementing asynchronous I/O in the kernel and also implementing a new caching mechanism using pre-registered buffers). See also: http://sourceforge.net/mailarchive/forum.php?forum_name=scst-devel&viewmonth=20080... Bart Van Assche. --
Just because Solaris takes a particular design decision doesn't automatically make it the right course of action. Microsoft once pulled huge gobs of the C library and their windowing system into the kernel in the name of efficiency. It proved not only to be less efficient, but also to degrade their security model. Deciding what lives in userspace and what should be in the kernel lies at the very heart of architectural decisions. However, the argument that "it should be in the kernel because that would make it faster" is pretty much a discredited one. To prevail on that argument, you have to demonstrate that there's no way to enable user space to do the same thing at the same speed. Further, it was the same argument used the last time around when the STGT vs SCST investigation was done. Your own results on non-IB networks show that both architectures perform at the same speed. That tends to support the conclusion that there's something specific about IB that needs to be tweaked or improved for STGT to get it to perform correctly. Furthermore, if you have already decided before testing that SCST is right and that STGT is wrong based on the architectures, it isn't exactly going to increase my confidence in your measurement methodology These are both features being independently worked on, are they not? Even if they weren't, the combination of the size of SCST in kernel plus the problem of having to find a migration path for the current STGT users still looks to me to involve the greater amount of work. James --
On Jan 30, 2008 5:22 PM, James Bottomley You should know that given two different implementations in software of the same communication protocol, differences in latency and throughput become more visible as the network latency gets lower and the throughput gets higher. That's why conclusions can only be drawn from the InfiniBand numbers, and not from the 1 Gbit/s Ethernet numbers. Assuming that there is something I did not draw any conclusions from the architecture -- the only data I based my My proposal was to have both the SCST kernel code and the STGT kernel code in the mainstream Linux kernel. This would make it easier for current STGT users to evaluate SCST. It's too early to choose one of the two projects -- this choice can be made later on. Bart. --
I don't want to be mean, but does anyone actually use STGT in production? Seriously? In the latest development version of STGT, it's only possible to stop the tgtd target daemon using KILL / 9 signal - which also means all iSCSI initiator connections are corrupted when tgtd target daemon is started again (kernel upgrade, target daemon upgrade, server reboot etc.). Imagine you have to reboot all your NFS clients when you reboot your NFS server. Not only that - your data is probably corrupted, or at least the filesystem deserves checking... -- Tomasz Chmielewski http://wpkg.org --
On Tue, 05 Feb 2008 08:14:01 +0100 I don't know what "iSCSI initiator connections are corrupted" mean. But if you reboot a server, how can an iSCSI target --
The problem with tgtd is that you can't start it (configured) in an "atomic" way. Usually, one will start tgtd and it's configuration in a script (I replaced some parameters with "..." to make it shorter and more readable): tgtd tgtadm --op new ... tgtadm --lld iscsi --op new ... However, this won't work - tgtd goes immediately in the background as it is still starting, and the first tgtadm commands will fail: # bash -x tgtd-start + tgtd + tgtadm --op new --mode target ... tgtadm: can't connect to the tgt daemon, Connection refused tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected + tgtadm --lld iscsi --op new --mode account ... tgtadm: can't connect to the tgt daemon, Connection refused tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected + tgtadm --lld iscsi --op bind --mode account --tid 1 ... tgtadm: can't find the target + tgtadm --op new --mode logicalunit --tid 1 --lun 1 ... tgtadm: can't find the target + tgtadm --op bind --mode target --tid 1 -I ALL tgtadm: can't find the target + tgtadm --op new --mode target --tid 2 ... + tgtadm --op new --mode logicalunit --tid 2 --lun 1 ... + tgtadm --op bind --mode target --tid 2 -I ALL OK, if tgtd takes longer to start, perhaps it's a good idea to sleep a second right after tgtd? tgtd sleep 1 tgtadm --op new ... tgtadm --lld iscsi --op new ... No, it is not a good idea - if tgtd listens on port 3260 *and* is unconfigured yet, any reconnecting initiator will fail, like below: end_request: I/O error, dev sdb, sector 7045192 Buffer I/O error on device sdb, logical block 880649 lost page write due to I/O error on sdb Aborting journal on device sdb. ext3_abort called. EXT3-fs error (device sdb): ext3_journal_start_sb: Detected aborted journal Remounting filesystem read-only end_request: I/O error, dev sdb, sector 7045880 Buffer I/O error on device sdb, logical block 880735 lost page write due to I/O error on ...
this should be a easy fix. start tgtd, get port setup ready in forked process, then signal its parent that ready to quit. or set port ready in this is another easy fix. tgtd started with unconfigured status and then a tgtadm can configure it and turn it into ready status. those are really minor usability issue. ( i know it is painful for user, i agree) the major problem here is to discuss in architectural wise, which one is better... linux kernel should have one implementation that is good from -- Ming Zhang @#$%^ purging memory... (*!% http://blackmagic02881.wordpress.com/ http://www.linkedin.com/in/blackmagic02881 -------------------------------------------- --
On Tue, 05 Feb 2008 17:07:07 +0100 Thanks for the details. So the way to stop the daemon is not related with your problem. It's easily fixable. Can you start a new thread about this on stgt-devel mailing list? When we agree on the interface to start the I don't know how many people use stgt in a production environment but I'm not sure that this problem prevents many people from using it in a production environment. You want to reboot a server running target devices while initiators connect to it. Rebooting the target server behind the initiators seldom works. System adminstorators in my workplace reboot storage devices once a year and tell us to shut down the initiator machines that use them before that. --
Don't know if matters, but in my setup (iscsi on top of drbd+heartbeat) rebooting the primary server doesn't affect my iscsi traffic, SCST correctly manages stop/crash, by sending unit attention to clients on reconnect. Drbd+heartbeat correctly manages those things too. Still from an end-user POV, i was able to reboot/survive a crash only with SCST, IETD still has reconnect problems and STGT are even worst. Regards, --matteo --
On Tue, 05 Feb 2008 18:09:15 +0100 Please tell us on stgt-devel mailing list if you see problems. We will try to fix them. Thanks, --
The TCP connection will drop, remember that the TCP connection state for one side has completely vanished. Depending on iSCSI/iSER ErrorRecoveryLevel that is set, this will mean: 1) Session Recovery, ERL=0 - Restarting the entire nexus and all connections across all of the possible subnets or comm-links. All outstanding un-StatSN acknowledged commands will be returned back to the SCSI subsystem with RETRY status. Once a single connection has been reestablished to start the nexus, the CDBs will be resent. 2) Connection Recovery, ERL=2 - CDBs from the failed connection(s) will be retried (nothing changes in the PDU) to fill the iSCSI CmdSN ordering gap, or be explictly retried with TMR TASK_REASSIGN for ones already acknowledged by the ExpCmdSN that are returned to the initiator in FYI, the LIO code also supports rmmoding iscsi_target_mod while at full 10 Gb/sec speed. I think it should be a requirement to be able to control per initiator, per portal group, per LUN, per device, per HBA in the design without restarting any other objects. --
James, Although the performance difference between STGT and SCST is apparent, this isn't the only point why SCST is better. I've already written about it many times in various mailing lists, but let me summarize it one more time here. As you know, almost all kernel parts can be done in user space, including all the drivers, networking, I/O management with block/SCSI initiator subsystem and disk cache manager. But does it mean that currently Linux kernel is bad and all the above should be (re)done in user space instead? I believe, not. Linux isn't a microkernel for very pragmatic reasons: simplicity and performance. So, additional important point why SCST is better is simplicity. For SCSI target, especially with hardware target card, data are came from kernel and eventually served by kernel, which does actual I/O or getting/putting data from/to cache. Dividing requests processing between user and kernel space creates unnecessary interface layer(s) and effectively makes the requests processing job distributed with all its complexity and reliability problems. From my point of view, having such distribution, where user space is master side and kernel is slave is rather wrong, because: 1. It makes kernel depend from user program, which services it and provides for it its routines, while the regular paradigm is the opposite: kernel services user space applications. As a direct consequence from it that there is no real protection for the kernel from faults in the STGT core code without excessive effort, which, no surprise, wasn't currently done and, seems, is never going to be done. So, on practice debugging and developing under STGT isn't easier, than if the whole code was in the kernel space, but, actually, harder (see below why). 2. It requires new complicated interface between kernel and user spaces that creates additional maintenance and debugging headaches, which don't exist for kernel only code. Linus Torvalds some time ago perfectly ...
So, James, what is your opinion on the above? Or the overall SCSI target project simplicity doesn't matter much for you and you think it's fine to duplicate Linux page cache in the user space to keep the in-kernel part of the project as small as possible? Vlad --
It's too early to draw conclusions about performance. I'm currently performing more measurements, and the results are not easy to interpret. My plan is to measure the following: * Setup: target with RAM disk of 2 GB as backing storage. * Throughput reported by dd and xdd (direct I/O). * Transfers with dd/xdd in units of 1 KB to 1 GB (the smallest transfer size that can be specified to xdd is 1 KB). * Target SCSI software to be tested: IETD iSCSI via IPoIB, STGT iSCSI via IPoIB, STGT iSER, SCST iSCSI via IPoIB, SCST SRP, LIO iSCSI via IPoIB. The reason I chose dd/xdd for these tests is that I want to measure the performance of the communication protocols, and that I am assuming that this performance can be modeled by the following formula: (transfer time in s) = (transfer setup latency in s) + (transfer size in MB) / (bandwidth in MB/s). Measuring the time needed for transfers with varying block size allows to compute the constants in the above formula via linear regression. One difficulty I already encountered is that the performance of the Linux IPoIB implementation varies a lot under high load (http://bugzilla.kernel.org/show_bug.cgi?id=9883). Another issue I have to look further into is that dd and xdd report different results for very large block sizes (> 1 MB). Bart Van Assche. --
Be aware that xdd reports 1 MB as 1000000, not 1048576. Though, it looks
like dd is the same, so that's probably not helpful. Also, make sure
you're passing {i,o}flag=direct to dd if you're using -dio in xdd to be
sure you are comparing apples to apples.
--
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office
--
It isn't fully correct, you forgot about link latency. More correct one is: (transfer time) = (transfer setup latency on both initiator and target, consisting from software processing time, including memory copy, if necessary, and PCI setup/transfer time) + (transfer size)/(bandwidth) + (link latency to deliver request for READs or status for WRITES) + (2*(link latency) to deliver R2T/XFER_READY request in case of WRITEs, if necessary (e.g. iSER for small transfers might not need it, but SRP most likely always needs it)). Also you should note that it's correct only in case of single threaded workloads with one outstanding command at time. For other workloads it depends from how well they manage to keep the "link" full in interval from (transfer size)/(transfer time) to Look at /proc/scsi_tgt/sgv (for SCST) and you will see, which transfer sizes are actually used. Initiators don't like sending big requests and often split them on smaller ones. Look at this message as well, it might be helpful: --
Regarding the performance tests I promised to perform: although until
now I only have been able to run two tests (STGT + iSER versus SCST +
SRP), the results are interesting. I will run the remaining test cases
during the next days.
About the test setup: dd and xdd were used to transfer 2 GB of data
between an initiator system and a target system via direct I/O over an
SDR InfiniBand network (1GB/s). The block size varied between 512
bytes and 1 GB, but was always a power of two.
Expected results:
* The measurement results are consistent with the numbers I published earlier.
* During data transfers all data is transferred in blocks between 4 KB
and 32 KB in size (according to the SCST statistics).
* For small and medium block sizes (<= 32 KB) transfer times can be
modeled very well by the following formula: (transfer time) = (setup
latency) + (bytes transferred)/(bandwidth). The correlation numbers
are very close to one.
* The latency and bandwidth parameters depend on the test tool (dd
versus xdd), on the kind of test performed (reading versus writing),
on the SCSI target and on the communication protocol.
* When using RDMA (iSER or SRP), SCST has a lower latency and higher
bandwidth than STGT (results from linear regression for block sizes <=
32 KB):
Test Latency(us) Bandwidth (MB/s) Correlation
STGT+iSER, read, dd 64 560 0.999995
STGT+iSER, read, xdd 65 556 0.999994
STGT+iSER, write, dd 53 394 0.999971
STGT+iSER, write, xdd 54 445 0.999959
SCST+SRP, read, dd 39 657 0.999983
SCST+SRP, read, xdd 41 668 0.999987
SCST+SRP, write, dd 52 449 0.999962
SCST+SRP, write, xdd 52 516 0.999977
Results that I did not expect:
* A block transfer size of 1 MB is not enough to measure the maximal
throughput. The maximal throughput is only reached at much ...Block transfer sizes over about 64kB are totally irrelevant for 99% of all people. Don't even bother testing anything more. Yes, bigger transfers happen, but a lot of common loads have *smaller* transfers than 64kB. So benchmarks that try to find "theoretical throughput" by just making big transfers should just be banned. They give numbers, yes, but the numbers are pointless. Linus --
The answers were pretty much contained here http://marc.info/?l=linux-scsi&m=120164008302435 and here: http://marc.info/?l=linux-scsi&m=120171067107293 Weren't they? James --
No, sorry, it doesn't look so for me. They are about performance, but I'm asking about the overall project's architecture, namely about one part of it: simplicity. Particularly, what do you think about duplicating Linux page cache in the user space to have zero-copy cached I/O? Or can you suggest another architectural solution for that problem in the STGT's approach? Vlad --
Isn't that an advantage of a user space solution? It simply uses the backing store of whatever device supplies the data. That means it takes advantage of the existing mechanisms for caching. James --
No, please reread this thread, especially this message: http://marc.info/?l=linux-kernel&m=120169189504361&w=2. This is one of the advantages of the kernel space implementation. The user space implementation has to have data copied between the cache and user space buffer, but the kernel space one can use pages in the cache directly, without extra copy. Vlad --
Well, you've said it thrice (the bellman cried) but that doesn't make it true. The way a user space solution should work is to schedule mmapped I/O from the backing store and then send this mmapped region off for target I/O. For reads, the page gather will ensure that the pages are up to date from the backing store to the cache before sending the I/O out. For writes, You actually have to do a msync on the region to get the data secured to the backing store. You also have to pull tricks with the mmap region in the case of writes to prevent useless data being read in from the backing store. However, none of this involves data copies. James --
James, have you checked how fast is mmaped I/O if work size > size of RAM? It's several times slower comparing to buffered I/O. It was many times discussed in LKML and, seems, VM people consider it unavoidable. So, using mmaped IO isn't an option for high performance. Plus, mmaped IO isn't an option for high reliability requirements, since it doesn't Can you be more exact and specify what kind of tricks should be done for --
Erm, but if you're using the case of work size > size of RAM, you'll find buffered I/O won't help because you don't have the memory for I think you'll find it does ... the page gather returns -EFAULT if there's an I/O error in the gathered region. msync does something Actually, just avoid touching it seems to do the trick with a recent kernel. James --
Err, to whom return? If you try to read from a mmaped page, which can't be populated due to I/O error, you will get SIGBUS or SIGSEGV, I don't remember exactly. It's quite tricky to get back to the faulted command from the signal handler. Or do you mean mmap(MAP_POPULATE)/munmap() for each command? Do you --
So in an out of memory situation the buffers you don't have are a lot I meant from user space ... the writes are done inside the kernel. However, as Linus has pointed out, this discussion is getting a bit off topic. There's no actual evidence that copy problems are causing any performatince issues issues for STGT. In fact, there's evidence that they're not for everything except IB networks. James --
There isn't OOM in both cases. Just pages reclamation/readahead work Sure, the mmap() approach agreed to be unpractical, but could you elaborate more on this anyway, please? I'm just curious. Do you think about implementing a new syscall, which would put pages with data in the No, that isn't off topic. We've just proved that there is no good way to implement zero-copy cached I/O for STGT. I see the only practical way for that, proposed by FUJITA Tomonori some time ago: duplicating Linux The zero-copy cached I/O has not yet been implemented in SCST, I simply so far have not had time for that. Currently SCST performs better STGT, because of simpler processing path and less context switches per command. Memcpy() speed on modern systems is about the same as throughput of 20Gbps link (1600MB/s), so when the zero-copy will be --
No, it has to do with the way invalidation occurs. When you mmap a region from a device or file, the kernel places page translations for that region into your vm_area. The regions themselves aren't backed until faulted. For write (i.e. incoming command to target) you specify the write flag and send the area off to receive the data. The gather, expecting the pages to be overwritten, backs them with pages marked dirty but doesn't fault in the contents (unless it already exists in the page cache). The kernel writes the data to the pages and the dirty pages go back to the user. msync() flushes them to the device. The disadvantage of all this is that the handle for the I/O if you will is a virtual address in a user process that doesn't actually care to see the data. non-x86 architectures will do flushes/invalidates on this Well, there's no real evidence that zero copy or lack of it is a problem yet. James --
I more or less see, thanks. But (1) pages still needs to be mmaped to the user space process before the data transmission, i.e. they must be zeroed before being mmaped, which isn't much faster, than data copy, and (2) I suspect, it would be hard to make it race free, e.g. if another The performance improvement from zero copy can be easily estimated, knowing the link throughput and data copy throughput, which are about the same for 20Gbps links (I did that few e-mail ago). Vlad --
Since the focus of this thread shifted somewhat in the last few messages, I'll try to summarize what has been discussed so far: - There was a number of participants who joined this discussion spontaneously. This suggests that there is considerable interest in networked storage and iSCSI. - It has been motivated why iSCSI makes sense as a storage protocol (compared to ATA over Ethernet and Fibre Channel over Ethernet). - The direct I/O performance results for block transfer sizes below 64 KB are a meaningful benchmark for storage target implementations. - It has been discussed whether an iSCSI target should be implemented in user space or in kernel space. It is clear now that an implementation in the kernel can be made faster than a user space implementation (http://kerneltrap.org/mailarchive/linux-kernel/2008/2/4/714804). Regarding existing implementations, measurements have a.o. shown that SCST is faster than STGT (30% with the following setup: iSCSI via IPoIB and direct I/O block transfers with a size of 512 bytes). - It has been discussed which iSCSI target implementation should be in the mainstream Linux kernel. There is no agreement on this subject yet. The short-term options are as follows: 1) Do not integrate any new iSCSI target implementation in the mainstream Linux kernel. 2) Add one of the existing in-kernel iSCSI target implementations to the kernel, e.g. SCST or PyX/LIO. 3) Create a new in-kernel iSCSI target implementation that combines the advantages of the existing iSCSI kernel target implementations (iETD, STGT, SCST and PyX/LIO). As an iSCSI user, I prefer option (3). The big question is whether the various storage target authors agree with this ? Bart Van Assche. --
why should linux as an iSCSI target be limited to passthrough to a SCSI device. the most common use of this sort of thing that I would see is to load up a bunch of 1TB SATA drives in a commodity PC, run software RAID, and then export the resulting volume to other servers via iSCSI. not a 'real' SCSI device in sight. As far as how good a standard iSCSI is, at this point I don't think it really matters. There are too many devices and manufacturers out there that implement iSCSI as their storage protocol (from both sides, offering storage to other systems, and using external storage). Sometimes the best technology doesn't win, but Linux should be interoperable with as much as possible and be ready to support the winners and the loosers in technology options, for as long as anyone chooses to use the old equipment (after all, we support things like Arcnet networking, which lost to Ethernet many years ago) David Lang --
David, your question surprises me a lot. From where have you decided that SCST supports only pass-through backstorage? Does the RAM disk, which Bart has been using for performance tests, look like a SCSI device? SCST supports all backstorage types you can imagine and Linux kernel --
I was responding to the start of item #2 that I left in the quote above. it asn't saying that SCST didn't support that, but was stating that any implementation of a iSCSI target should use the SCSI framework. I read this to mean that this would only be able to access things that the SCSI framework can access, and that would not be things like ramdisks, raid arrays, etc. David Lang --
<nod> I don't think anyone is saying it should be. It makes sense that the more mature SCSI engines that have working code will be providing alot supports very SCSI specific target mode hardware, including software target mode forks of other kernel code. This code for the target mode pSCSI, FC and SAS control paths (more for the state machines, that CDB emulation) that will most likely never need to be emulated on non SCSI target engine. SCST has support for the most SCSI fabric protocols of the group (although it is lacking iSER) while the LIO-SE only supports traditional iSCSI using Linux/IP (this means TCP, SCTP and IPv6). The design of LIO-SE was to make every iSCSI initiator that sends SCSI CDBs and data to talk to every potential device in the Linux storage stack on the largest amount of hardware architectures possible. Most of the iSCSI Initiators I know (including non Linux) do not rely on heavy SCSI task management, and I think this would be a lower priority item to get real SCSI specific recovery in the traditional iSCSI target for users. Espically things like SCSI target mode queue locking (affectionally called Auto Contingent Allegiance) make no sense for I recently moved the last core LIO target machine from a hardware RAID5 to MD RAID6 with struct block_device exported LVM objects via Linux/iSCSI to PVM and HVM domains, and I have been very happy with the results. Being able to export any physical or virtual storage object from whatever layer makes sense for your particular case. This applies to both block and file level access. For example, making an iSCSI Initiator and Target run in the most limited in environments places where NAS (espically userspace server side) would have a really hard time fitting, has always been a requirement. You can imagine a system with a smaller amount of memory (say 32MB) having a difficult time doing I/O to any amount of NAS clients. If are talking about memory required to get best performance, using kernel level ...
Sorry, it isn't correct. ACA provides possibility to lock commands queue in case of CHECK CONDITION, so allows to keep commands execution order in case of errors. CmdSN keeps commands execution order only in case of success, in case of error the next queued command will be executed immediately after the failed one, although application might require to have all subsequent after the failed one commands aborted. Think about journaled file systems, for instance. Also ACA allows to retry the failed command and then resume the queue. Vlad --
Fair enough. The point I was making is that I have never actually seen an iSCSI Initiator use ACA functionality (I don't believe that the Linux SCSI Ml implements this), or actually generate a CLEAR_ACA task management request. --
If I understood the above correctly, regarding a kernel space iSCSI target implementation, only LIO-SE and SCST should be considered. What I know today about these Linux iSCSI target implementations is as follows: * SCST performs slightly better than LIO-SE, and LIO-SE performs slightly better than STGT (both with regard to latency and with regard to bandwidth). * The coding style of SCST is closer to the Linux kernel coding style than the coding style of the LIO-SE project. * The structure of SCST is closer to what Linus expects than the structure of LIO-SE (i.e., authentication handled in userspace, data transfer handled by the kernel -- LIO-SE handles both in kernel space). * Until now I did not encounter any strange behavior in SCST. The issues I encountered with LIO-SE are being resolved via the LIO-SE mailing list (http://groups.google.com/group/linux-iscsi-target-dev). It would take too much effort to develop a new kernel space iSCSI target from scratch -- we should start from either LIO-SE or SCST. My opinion is that the best approach is to start with integrating SCST in the mainstream kernel, and that the more advanced features from LIO-SE that are not yet in SCST can be ported from LIO-SE to the SCST framework. Nicholas, do you think the structure of SCST is powerful enough to be extended with LIO-SE's powerful features like ERL-2 ? Bart Van Assche. --
I think the other data point here would be that final target design needs to be as generic as possible. Generic in the sense that the engine eventually needs to be able to accept NDB and other ethernet based target mode storage configurations to an abstracted device object (struct scsi_device, struct block_device, or struct file) just as it would for an IP Storage based request. We know that NDB and *oE will have their own naming and discovery, and the first set of IO tasks to be completed would be those using (iscsi_cmd_t->cmd_flags & ICF_SCSI_DATA_SG_IO_CDB) in iscsi_target_transport.c in the current code. These are single READ_* and WRITE_* codepaths that perform DMA memory pre-proceessing in v2.9 LIO-SE. Also, being able to tell the engine to accelerate to DMA ring operation (say to underlying struct scsi_device or struct block_device) instead of fileio in some cases you will see better performance when using hardware (ie: not a underlying kernel thread queueing IO into block). But I have found FILEIO with sendpage with MD to be faster in single threaded tests than struct block_device. I am currently using IBLOCK for LVM for core LIO operation (which actually sits on software MD raid6). I do this because using submit_bio() with se_mem_t mapped arrays of struct scatterlist -> struct bio_vec can handle power failures properly, and not send back StatSN Acks to the Initiator who thinks that everything has already made it to disk. This is the case with doing IO to struct file in the kernel today without a kernel level O_DIRECT. Also for proper kernel-level target mode support, using struct file with O_DIRECT for storage blocks and emulating control path CDBS is one of the work items. This can be made generic or obtained from the underlying storage object (anything that can be exported from LIO Subsystem TPI) For real hardware (struct scsi_device in just about all the cases these days). Last time I looked this was due to fs/direct-io.c:dio_refill_pages() using ...
Is there an open iSCSI Target implementation which does NOT issue commands to sub-target devices via the SCSI mid-layer, but bypasses it completely? Luben --
What do you mean? To call directly low level backstorage SCSI drivers --
Yes, that's what I meant. Just curious. Thanks, Luben --
Hi Luben, I am guessing you mean futher down the stack, which I don't know this to be the case. Going futher up the layers is the design of v2.9 LIO-SE. There is a diagram explaining the basic concepts from a 10,000 foot level. http://linux-iscsi.org/builds/user/nab/storage-engine-concept.pdf Note that only traditional iSCSI target is currently implemented in v2.9 LIO-SE codebase in the list of target mode fabrics on left side of the layout. The API between the protocol headers that does encoding/decoding target mode storage packets is probably the least mature area of the LIO stack (because it has always been iSCSI looking towards iSER :). I don't know who has the most mature API between the storage engine and target storage protocol for doing this between SCST and STGT, I am guessing SCST because of the difference in age of the projects. Could someone be so kind to fill me in on this..? Also note, the storage engine plugin for doing userspace passthrough on the right is also currently not implemented. Userspace passthrough in this context is an target engine I/O that is enforcing max_sector and sector_size limitiations, and encodes/decodes target storage protocol packets all out of view of userspace. The addressing will be completely different if we are pointing SE target packets at non SCSI target ports in userspace. --
SCST uses scsi_execute_async_fifo() function to submit commands to SCSI devices in the pass-through mode. This function is slightly modified version of scsi_execute_async(), which submits requests in FIFO order instead of LIFO as scsi_execute_async() does (so with scsi_execute_async() they are executed in the reverse order). --
The LIO-SE PSCSI Plugin also depends on scsi_execute_async() for builds on >= 2.6.18. Note in the core LIO storage engine code (would be iscsi_target_transport.c), there is no subsystem dependence logic. The LIO-SE API is what allows the SE plugins to remain simple and small: -rw-r--r-- 1 root root 35008 2008-02-02 03:25 iscsi_target_pscsi.c -rw-r--r-- 1 root root 7537 2008-02-02 17:27 iscsi_target_pscsi.h -rw-r--r-- 1 root root 18269 2008-02-04 02:23 iscsi_target_iblock.c -rw-r--r-- 1 root root 6834 2008-02-04 02:25 iscsi_target_iblock.h -rw-r--r-- 1 root root 30611 2008-02-02 03:25 iscsi_target_file.c -rw-r--r-- 1 root root 7833 2008-02-02 17:27 iscsi_target_file.h -rw-r--r-- 1 root root 35154 2008-02-02 04:01 iscsi_target_rd.c -rw-r--r-- 1 root root 9900 2008-02-02 17:27 iscsi_target_rd.h It also means that the core LIO-SE code does not have to change when the subsystem APIs change. This has been important in the past for the project, but for upstream code, probably would not make a huge difference. --nab --
mmap'ing may avoid the copy, but the overhead of a mmap operation is quite often much *bigger* than the overhead of a copy operation. Please do not advocate the use of mmap() as a way to avoid memory copies. It's not realistic. Even if you can do it with a single "mmap()" system call (which is not at all a given, considering that block devices can easily be much larger than the available virtual memory space), the fact is that page table games along with the fault (and even just TLB miss) overhead is easily more than the cost of copying a page in a nice streaming manner. "data copies" is irrelevant. The only thing that matters is performance. And if avoiding data copies is more costly (or even of a similar cost) than the copies themselves would have been, there is absolutely no upside, and only downsides due to extra complexity. If you want good performance for a service like this, you really generally *do* need to in kernel space. You can play games in user space, but you're fooling yourself if you think you can do as well as doing it in the kernel. And you're *definitely* fooling yourself if you think mmap() solves performance issues. "Zero-copy" does not equate to "fast". Memory speeds may be slower that core CPU speeds, but not infinitely so! (That said: there *are* alternatives to mmap, like "splice()", that really do potentially solve some issues without the page table and TLB overheads. But while splice() avoids the costs of paging, I strongly suspect it would still have easily measurable latency issues. Switching between user and kernel space multiple times is definitely not going to be free, although it's probably not a huge issue if you have big enough requests). Linus --
Sorry ... this is really just a discussion of how something (zero copy) could be done, rather than an implementation proposal. (I'm not actually planning to make the STGT people do anything ... although investigating splice does sound interesting). Right at the moment, STGT seems to be performing just fine on measurements up to gigabit networks. There are suggestions that there may be a problem on 8G IB networks, but it's not definitive yet. I'm already on record as saying I think the best fix for IB networks is just to reduce the context switches by increasing the transfer size, but the infrastructure to allow that only just went into git head. James --
The iSER spec (RFC-5046) quotes the following in the TCP case for direct data placement: " Out-of-order TCP segments in the Traditional iSCSI model have to be stored and reassembled before the iSCSI protocol layer within an end node can place the data in the iSCSI buffers. This reassembly is required because not every TCP segment is likely to contain an iSCSI header to enable its placement, and TCP itself does not have a built-in mechanism for signaling Upper Level Protocol (ULP) message boundaries to aid placement of out-of-order segments. This TCP reassembly at high network speeds is quite counter-productive for the following reasons: wasted memory bandwidth in data copying, the need for reassembly memory, wasted CPU cycles in data copying, and the general store-and-forward latency from an application perspective." While this does not have anything to do directly with the kernel vs. user discussion for target mode storage engine, the scaling and latency case is easy enough number of years, I would be inclined to believe this is true for software and hardware data-path cases. The benefits of moving various control statemachines for something like say traditional iSCSI to userspace has always been debateable. The most obvious ones are things like authentication, espically if something more complex than CHAP are the obvious case for userspace. However, I have thought recovery for failures caused from communication path (iSCSI connections) or entire nexuses (iSCSI sessions) failures was very problematic to expect to have to potentially push down IOs state to userspace. Keeping statemachines for protocol and/or fabric specific statemachines (CSM-E and CSM-I from connection recovery in iSCSI and iSER are the Most of the SCSI OS storage subsystems that I have worked with in the context of iSCSI have used 256 * 512 byte setctor requests, which the default traditional iSCSI PDU data payload (MRDSL) being 64k to hit the sweet spot with crc32c ...
Then again, having some data-path for software and hardware bulk IO operation of storage fabric protocol / statemachine in userspace would be really interesting for something like an SPU enabled engine for the Cell Broadband Architecture. --nab --
I would like to point out that while I think there is no question that the basic data transfer engine would perform better in kernel space, there stll *are* questions whether - iSCSI is relevant enough for us to even care ... - ... and the complexity is actually worth it. That said, I also tend to believe that trying to split things up between kernel and user space is often more complex than just keeping things in one place, because the trade-offs of which part goes where wll inevitably be wrong in *some* area, and then you're really screwed. So from a purely personal standpoint, I'd like to say that I'm not really interested in iSCSI (and I don't quite know why I've been cc'd on this whole discussion) and think that other approaches are potentially *much* better. So for example, I personally suspect that ATA-over-ethernet is way better than some crazy SCSI-over-TCP crap, but I'm biased for simple and low-level, and against those crazy SCSI people to begin with. So take any utterances of mine with a big pinch of salt. Historically, the only split that has worked pretty well is "connection initiation/setup in user space, actual data transfers in kernel space". Pure user-space solutions work, but tend to eventually be turned into kernel-space if they are simple enough and really do have throughput and latency considerations (eg nfsd), and aren't quite complex and crazy enough to have a large impedance-matching problem even for basic IO stuff (eg samba). And totally pure kernel solutions work only if there are very stable standards and no major authentication or connection setup issues (eg local disks). So just going by what has happened in the past, I'd assume that iSCSI would eventually turn into "connecting/authentication in user space" with "data transfers in kernel space". But only if it really does end up mattering enough. We had a totally user-space NFS daemon for a long time, and it was perfectly fine until people really ...
surely aoe is better than iscsi almost on performance because of the lesser protocol stack: iscsi -> scsi - ip - eth aoe -> ata - eth but surely iscsi is more a standard than aoe and is more actively used by real-world . Other really useful feature are that: - iscsi is capable to move to a ip based san scsi devices by routing that ( i've some tape changer routed by scst to some system that don't have other way to see a tape). - because it work on the ip layer it can be routed between long distance , so having needed bandwidth you can have a really remote block device spoking a standard protocol between non ethereogenus systems. - iscsi is now the cheapest san avaible. bye, marco. --
The generic target mode storage engine discussion quickly goes to transport specific scenarios. With so much interest in the SCSI transports, in particuarly iSCSI, there are lots of devs, users, and Having the non SCSI target mode transports use the same data IO path as the SCSI ones to SCSI, BIO, and FILE subsystems is something that can easily be agreed on. Also having to emulate the non SCSI control paths in a non generic matter to a target mode engine has to suck (I don't know what AoE does for that now, considering that this is going down to libata or real SCSI hardware in some cases. There are some of the more arcane task management functionality in SCSI (ACA anyone?) that even generic SCSI target mode engines do not use, and only seem to make endlessly complex implement and emulate. But aside from those very SCSI hardware specific cases, having a generic method to use something like ABORT_TASK or LUN_RESET for a target mode engine (along with the data path to all of the subsystems) would be Well, having no obvious preconception (well, aside from the email address), I am of the mindset than the iSCSI people are the LEAST crazy said crazy SCSI people. Some people (usually least crazy iSCSI standards folks) say that FCoE people are crazy. Being one of the iSCSI people I am kinda obligated to agree, but the technical points are really solid, and have been so for over a decade. They are listed here for those who are interested: Thanks for putting this into an historical perspective. Also it is interesting to note that the iSCSI spec (RFC-3720) was ratified in April 2004, so it will be going on 4 years soon, which pre-RFC products first going out in 2001 (yikes!). In my experience, the iSCSI interopt amongst implementations (espically between different OSes) has been stable since about late 2004, early 2005, with interopt between OS SCSI subsystems (espically talking to non SCSI hardware) being the slower of the two. --nab --
On Mon, Feb 04, 2008 at 11:44:31AM -0800, Linus Torvalds wrote: I'd assumed the move was primarily because of the difficulty of getting correct semantics on a shared filesystem--if you're content with NFS-only access to your filesystem, then you can probably do everything in userspace, but once you start worrying about getting stable filehandles, consistent file locking, etc., from a real disk filesystem with local users, then you require much closer cooperation from the kernel. And I seem to recall being told that sort of thing was the motivation more than performance, but I wasn't there (and I haven't seen performance comparisons). --b. --
.. not even shared. It was hard to get correct semantics full stop. Which is a traditional problem. The thing is, the kernel always has some internal state, and it's hard to expose all the semantics that the kernel knows about to user space. So no, performance is not the only reason to move to kernel space. It can easily be things like needing direct access to internal data queues (for a iSCSI target, this could be things like barriers or just tagged commands - yes, you can probably emulate things like that without access to the actual IO queues, but are you sure the semantics will be entirely right? The kernel/userland boundary is not just a performance boundary, it's an abstraction boundary too, and these kinds of protocols tend to break abstractions. NFS broke it by having "file handles" (which is not something that really exists in user space, and is almost impossible to emulate correctly), and I bet the same thing happens when emulating a SCSI target in user space. Maybe not. I _rally_ haven't looked into iSCSI, I'm just guessing there would be things like ordering issues. Linus --
<nod>. The iSCSI CDBs and write immediate, unsoliciated, or soliciated data payloads may be received out of order across communication paths (which may be going over different subnets) within the nexus, but the execution of the CDB to SCSI Target Port must be in the same order as it came down from the SCSI subsystem on the initiator port. In iSCSI and iSER terms, this is called Command Sequence Number (CmdSN) ordering, and is enforced within each nexus. The initiator node will be assigning the CmdSNs as the CDBs come down, and when communication paths fail, unacknowledged CmdSNs will be retried on a different communication path when using iSCSI/iSER connection recovery. Already acknowledged CmdSNs will be explictly retried using a iSCSI specific task management function called TASK_REASSIGN. This along with CSM-I and CSM-E statemachines are collectly known as ErrorRecoveryLevel=2 in iSCSI. Anyways, here is a great visual of a modern iSCSI Target processor and SCSI Target Engine. The CmdSN ordering is representd by the oval across across iSCSI connections going to various network portals groups on the left side of the diagram. Thanks Eddy Q! http://www.haifa.il.ibm.com/satran/ips/EddyQuicksall-iSCSI-in-diagrams/portal_groups.pdf --nab --
Well, speaking as a complete nutter who just finished the bare bones of an NFSv4 userland server[1]... it depends on your approach. If the userland server is the _only_ one accessing the data[2] -- i.e. the database server model where ls(1) shows a couple multi-gigabyte files or a raw partition -- then it's easy to get all the semantics right, including file handles. You're not racing with local kernel fileserving. Couple that with sendfile(2), sync_file_range(2) and a few other Linux-specific syscalls, and you've got an efficient NFS file server. It becomes a solution similar to Apache or MySQL or Oracle. I quite grant there are many good reasons to do NFS or iSCSI data path in the kernel... my point is more that "impossible" is just from one iSCSI and NBD were passe ideas at birth. :) Networked block devices are attractive because the concepts and implementation are more simple than networked filesystems... but usually you want to run some sort of filesystem on top. At that point you might as well run NFS or [gfs|ocfs|flavor-of-the-week], and ditch your networked block device (and associated complexity). iSCSI is barely useful, because at least someone finally standardized SCSI over LAN/WAN. But you just don't need its complexity if your filesystem must have its own authentication, distributed coordination, multiple-connection management code of its own. Jeff P.S. Clearly my NFSv4 server is NOT intended to replace the kernel one. It's more for experiments, and doing FUSE-like filesystem work. [1] http://linux.yyz.us/projects/nfsv4.html [2] well, outside of dd(1) and similar tricks... the same "going around its back" tricks that can screw up a mounted filesystem. --
It's not really simple in general even then. The problems come with file handles, and two big issues in particular: - handling a reboot (of the server) without impacting the client really does need a "look up by file handle" operation (which you can do by logging the pathname to filehandle translation, but it certainly gets problematic). - non-Unix-like filesystems don't necessarily have a stable "st_ino" field (ie it may change over a rename or have no meaning what-so-ever, things like that), and that makes trying to generate a filehandle really interesting for them. I do agree that it's possible - we obviously _did_ have a user-level NFSD for a long while, after all - but it's quite painful if you want to handle things well. Only allowing access through the NFSD certainly helps a lot, but still doesn't make it quite as trivial as you claim ;) Of course, I think you can make NFSv4 to use volatile filehandles instead of the traditional long-lived ones, and that really should avoid almost all of the problems with doing a NFSv4 server in user space. However, I'd expect there to be clients that don't do the whole volatile thing, or support the file handle becoming stale only at certain well-defined points (ie after renames, not at random reboot times). Linus --
Both of these are easily handled if the server is 100% in charge of managing the filesystem _metadata_ and data. That's what I meant by complete control. i.e. it not ext3 or reiserfs or vfat, its a block device or 1000GB file managed by a userland process. Doing it that way gives one a bit more freedom to tune the filesystem format directly. Stable inode numbers and filehandles are just easy as they are with ext3. I'm the filesystem format designer, after all. (run for your lives...) You do wind up having to roll your own dcache in userspace, though. A matter of taste in implementation, but it is not difficult... I've Nah, you're thinking about something different: a userland NFSD competing with other userland processes for access to the same files, while the kernel ultimately manages the filesystem metadata. Recipe for races and inequities, and it's good we moved away from that. I'm talking about where a userland process manages the filesystem metadata too. In a filesystem with a million files, ls(1) on the server will only show a single file: [jgarzik@core ~]$ ls -l /spare/fileserver-data/ total 70657116 Don't get me started on "volatile" versus "persistent" filehandles in NFSv4... groan. Jeff --
Oh ok. Yes, if you bring the filesystem into user mode too, then the problems go away - because now your NFSD can interact directly with the filesystem without any kernel/usermode abstraction layer rules in between. So that has all the same properties as moving NFSD entirely into the kernel. Linus --
Running a filesystem on top of iSCSI results in better performance than NFS, especially if the NFS client conforms to the NFS standard (=synchronous writes). By searching the web search for the keywords NFS, iSCSI and performance I found the following (6 years old) document: http://www.technomagesinc.com/papers/ip_paper.html. A quote from the conclusion: Our results, generated by running some of industry standard benchmarks, show that iSCSI significantly outperforms NFS for situations when performing streaming, database like accesses and small file transactions. Bart Van Assche. --
async performs better than sync... this is news? Furthermore, NFSv4 has not only async capability but delegation too (and RDMA if you like such things), so the comparison is not relevant to modern times. But a networked filesystem (note I'm using that term, not "NFS", from here on) is simply far more useful to the average user. A networked block device is a building block -- and a useful one. A networked filesystem is an immediately usable solution. For remotely accessing data, iSCSI+fs is quite simply more overhead than a networked fs. With iSCSI you are doing local VFS -> local blkdev -> network whereas a networked filesystem is local VFS -> network iSCSI+fs also adds new manageability issues, because unless the filesystem is single-computer (such as diskless iSCSI root fs), you still need to go across the network _once again_ to handle filesystem locking and coordination issues. There is no _fundamental_ reason why remote shared storage via iSCSI OSD is any faster than a networked filesystem. SCSI-over-IP has its uses. Absolutely. It needed to be standardized. But let's not pretend iSCSI is anything more than what it is. Its a bloated cat5 cabling standard :) Jeff --
There are use cases than can be solved better via iSCSI and a filesystem than via a network filesystem. One such use case is when deploying a virtual machine whose data is stored on a network server: in that case there is only one user of the data (so there are no locking issues) and filesystem and block device each run in another operating system: the filesystem runs inside the virtual machine and iSCSI either runs in the hypervisor or in the native OS. Bart Van Assche. --
Hence the diskless root fs configuration I referred to in multiple emails... whoopee, you reinvented NFS root with quotas :) Jeff --
Call me a sysadmin, but I find easier to plug in and keep in place an ethernet cable than these parallel scsi cables from hell. Every server has at least two ethernet ports by default, with rarely any surprises at the kernel level. Adding ethernet cards is inexpensive, and you pretty much never hear of compatibility problems between cards. So ethernet as a connection medium is really nice compared to scsi. Too bad iscsi is demented and ATAoE/NBD inexistant. Maybe external SAS will be nice, but I don't see it getting to the level of universality of ethernet any time soon. And it won't get the same amount of user-level compatibility testing in any case. OG. --
Indeed, at the end of the day iSCSI is a bloated cabling standard. :) It has its uses, but I don't see it as ever coming close to replacing direct-to-network (perhaps backed with local cachefs) filesystems... which is how all the hype comes across to me. Cheap "Lintel" boxes everybody is familiar with _are_ the storage appliances. Until mass-produced ATA and SCSI devices start shipping with ethernet connectors, anyway. Jeff --
Yes, there is something like that for SCSI target as well. It's a "local initiator" or "local nexus", see http://thread.gmane.org/gmane.linux.scsi/31288 and http://news.gmane.org/find-root.php?message_id=%3c463F36AC.3010207%40vlnb.net%3e for more info about that. In fact, existence of local nexus is one more point why SCST is better, than STGT, because for STGT it's pretty hard to support it (all locally generated commands would have to be passed through its daemon, which would be a total disaster for performance), while for SCST it can be done relatively simply. Vlad --
Current ATAoE isn't. It can't support NCQ. A variant that did NCQ and IP would probably trash iSCSI for latency if nothing else. Alan --
In the previous iSCSI vs. FCoE points (here is the link again): http://www.ietf.org/mail-archive/web/ips/current/msg02325.html the latency discussion is the one bit that is not mentioned. I always assumed that back then (as with today) the biggest issue was getting ethernet hardware, espically switching equipment down to the sub millisecond latency, and on par with what you would expect from 'real RDMA' hardware. In lowest of the low, say sub 10 ns latency, which is apparently possible with point to point on high-end 10 Gb/sec adapters today, it would be really interesting to know how much more latency would be expected between software iSCSI vs. *oE when we work our way back up the networking stack. Julo, do you have any idea on this..? --
Actually, there's also FCoE now ... which is essentially SCSI encapsulated in Fibre Channel Protocols (FCP) running over ethernet with Jumbo frames. It does the standard SCSI TCQ, so should answer all the latency pieces. Intel even has an implementation: http://www.open-fcoe.org/ I tend to prefer the low levels as well. The whole disadvantage for IP as regards iSCSI was the layers of protocols on top of it for addressing, authenticating, encrypting and finding any iSCSI device anywhere in the connected universe. I tend to see loss of routing from operating at the MAC level to be a nicely justifiable tradeoff (most storage networks tend to be hubbed or switched anyway). Plus an ethernet MAC with jumbo frames is a large framed nearly lossless medium, which is practically what FCP is expecting. If you really have to connect large remote sites ... well that's what tunnelling bridges are for. James --
Btw, while simple in-band discovery of iSCSI exists, the standards based
IP storage deployments (iSCSI and iFCP) use iSNS (RFC-4171) for
discovery and network fabric management, for things like sending state
change notifications when a particular network portal is going away so
that the initiator can bring up a different communication patch to a
Some of the points by Julo on the IPS TWG iSCSI vs. FCoE thread:
* the network is limited in physical span and logical span (number
of switches)
* flow-control/congestion control is achieved with a mechanism
adequate for a limited span network (credits). The packet loss
rate is almost nil and that allows FCP to avoid using a
transport (end-to-end) layer
* FCP she switches are simple (addresses are local and the memory
requirements cam be limited through the credit mechanism)
* The credit mechanisms is highly unstable for large networks
(check switch vendors planning docs for the network diameter
limits) – the scaling argument
* Ethernet has no credit mechanism and any mechanism with a
similar effect increases the end point cost. Building a
transport layer in the protocol stack has always been the
preferred choice of the networking community – the community
argument
* The "performance penalty" of a complete protocol stack has
always been overstated (and overrated). Advances in protocol
stack implementation and finer tuning of the congestion control
mechanisms make conventional TCP/IP performing well even at 10
Gb/s and over. Moreover the multicore processors that become
dominant on the computing scene have enough compute cycles
available to make any "offloading" possible as a mere code
restructuring exercise (see the stack reports from Intel, IBM
etc.)
* Building on a complete stack makes available a wealth of
operational and ...Another data point from the "The "performance penalty of a complete protocol stack has always been overstated (and overrated)" bullet above: "As a side argument – a performance comparison made in 1998 showed SCSI over TCP (a predecessor of the later iSCSI) to perform better than FCP at 1Gbs for block sizes typical for OLTP (4-8KB). That was what convinced us to take the path that lead to iSCSI – and we used plain vanilla x86 servers with plain-vanilla NICs and Linux (with similar measurements conducted on Windows)." --nab --
This email somehow didn't manage to make it to the list (I suspect
because it had html attachments).
James
---
From:
Julian Satran
<Julian_Satran@il.ibm.com>
To:
Nicholas A. Bellinger
<nab@linux-iscsi.org>
Cc:
Andrew Morton
<akpm@linux-foundation.org>, Alan
Cox <alan@lxorguk.ukuu.org.uk>, Bart
Van Assche
<bart.vanassche@gmail.com>, FUJITA
Tomonori
<fujita.tomonori@lab.ntt.co.jp>,
James Bottomley
<James.Bottomley@HansenPartnership.com>, ...
Subject:
Re: Integration of SCST in the
mainstream Linux kernel
Date:
Mon, 4 Feb 2008 21:31:48 -0500
(20:31 CST)
Well stated. In fact the "layers" above ethernet do provide the services
that make the TCP/IP stack compelling - a whole complement of services.
ALL services required (naming, addressing, discovery, security etc.) will
have to be recreated if you take the FcOE route. That makes good business
for some but not necessary for the users. Those services BTW are not on
the data path and are not "overhead".
The TCP/IP stack pathlength is decently low. What makes most
implementations poor is that they where naively extended in the SMP world.
Recent implementations (published) from IBM and Intel show excellent
performance (4-6 times the regular stack). I do not have unfortunately
latency numbers (as the community major stress has been throughput) but I
assume that RDMA (not necessarily hardware RDMA) and/or the use of
infiniband or latency critical applications - within clusters may be the
ultimate low latency solution. Ethernet has some inherent latency issues
(the bridges) that are inherited by anything on ethernet (FcOE included).
The IP protocol stack is not inherently slow but some implementations are
somewhat sluggish.
But instead of replacing them with new and half backed contraptions we
would be all better of improving what we have and ...AoE is truly a thing of beauty. It has a two/three page RFC (say no more!). But quite so... AoE is limited to MTU size, which really hurts. Can't really do tagged queueing, etc. iSCSI is way, way too complicated. It's an Internet protocol designed by storage designers, what do you expect? For years I have been hoping that someone will invent a simple protocol (w/ strong auth) that can transit ATA and SCSI commands and responses. Heck, it would be almost trivial if the kernel had a TLS/SSL implementation. Jeff --
Why would you want authorization? If you don't use IP (just ethernet framing), then 99% of the time the solution is to just trust the subnet. So most people would never want TLS/SSL, and the ones that *do* want it would probably also want IP routing, so you'd actually be better off with a separate higher-level bridging protocol rather than have TLS/SSL as part of the actual packet protocol. So don't add complexity. The beauty of ATA-over-ethernet is exactly that it's simple and straightforward. (Simple and straightforward is also nice for actually creating devices that are the targets of this. I just *bet* that an iSCSI target device probably needs two orders of magnitude more CPU power than a simple AoE thing that can probably be done in an FPGA with no real software at all). Whatever. We have now officially gotten totally off topic ;) Linus --
I fully agree. From one side, all that complexity is unavoidable for case of multiple connections per session, but for the regular case of one connection per session it must be a lot simpler. And now think about iSER, which brings iSCSI on the whole new complexity --
Actually, think about those multiple connections... we already had to implement fast-failover (and load bal) SCSI multi-pathing at a higher level. IMO that portion of the protocol is redundant: You need the same capability elsewhere in the OS _anyway_, if you are to support multi-pathing. Jeff --
I'm thinking about MC/S as about a way to improve performance using several physical links. There's no other way, except MC/S, to keep commands processing order in that case. So, it's really valuable property of iSCSI, although with a limited application. Vlad --
Greetings, I have always observed the case with LIO SE/iSCSI target mode (as well as with other software initiators we can leave out of the discussion for now, and congrats to the open/iscsi on folks recent release. :-) that execution core hardware thread and inter-nexus per 1 Gb/sec ethernet port performance scales up to 4x and 2x core x86_64 very well with MC/S). I have been seeing 450 MB/sec using 2x socket 4x core x86_64 for a number of years with MC/S. Using MC/S on 10 Gb/sec (on PCI-X v2.0 266mhz as well, which was the first transport that LIO Target ran on that was able to reach handle duplex ~1200 MB/sec with 3 initiators and MC/S. In the point to point 10 GB/sec tests on IBM p404 machines, the initiators where able to reach ~910 MB/sec with MC/S. Open/iSCSI was able to go a bit faster (~950 MB/sec) because it uses struct sk_buff directly. A good rule to keep in mind here while considering performance is that context switching overhead and pipeline <-> bus stalling (along with other legacy OS specific storage stack limitations with BLOCK and VFS with O_DIRECT, et al and I will leave out of the discussion for iSCSI and SE engine target mode) is that a initiator will scale roughly 1/2 as well as a target, given comparable hardware and virsh output. The software target case target case also depends, in great regard in many cases, if we are talking about something something as simple as doing contiguous DMA memory allocations in from a SINGLE kernel thread, and handling direction execution to a storage hardware DMA ring that may have not been allocated in the current kernel thread. In MC/S mode this breaks down to: 1) Sorting logic that handles pre execution statemachine for transport from local RDMA memory and OS specific data buffers. TCP application data buffer, struct sk_buff, or RDMA struct page or SG. This should be generic between iSCSI and iSER. 2) Allocation of said memory buffers to OS subsystem dependent code that can be queued up to these ...
Sorry, these where IBM p505 express (not p404, duh) which had a 2x socket 2x core POWER5 setup. These along with an IBM X-series machine) where the only ones available for PCI-X v2.0, and this probably is still the case. :-) Also, these numbers where with a ~9000 MTU (I don't recall what the hardware limit on the 10 Gb/sec switch lwas) doing direct struct iovec to preallocated struct page mapping for payload on the target side. This is known as RAMDISK_DR plugin in the LIO-SE. On the initiator, LTP disktest and O_DIRECT where used for direct to SCSI block device access. I can big up this paper if anyone is interested. --nab --
Hello Nicholas, Are you sure that the LIO-SE kernel module source code is ready for inclusion in the mainstream Linux kernel ? As you know I tried to test the LIO-SE iSCSI target. Already while configuring the target I encountered a kernel crash that froze the whole system. I can reproduce this kernel crash easily, and I reported it 11 days ago on the LIO-SE mailing list (February 4, 2008). One of the call stacks I posted shows a crash in mempool_alloc() called from jbd. Or: the crash is most likely the result of memory corruption caused by LIO-SE. Because I was curious to know why it took so long to fix such a severe crash, I started browsing through the LIO-SE source code. Analysis of the LIO-SE kernel module source code learned me that this crash is not a coincidence. Dynamic memory allocation (kmalloc()/kfree()) in the LIO-SE kernel module is complex and hard to verify. There are 412 memory allocation/deallocation calls in the current version of the LIO-SE kernel module source code, which is a lot. Additionally, because of the complexity of the memory handling in LIO-SE, it is not possible to verify the correctness of the memory handling by analyzing a single function at a time. In my opinion this makes the LIO-SE source code hard to maintain. Furthermore, the LIO-SE kernel module source code does not follow conventions that have proven their value in the past like grouping all error handling at the end of a function. As could be expected, the consequence is that error handling is not correct in several functions, resulting in memory leaks in case of an error. Some examples of functions in which error handling is clearly incorrect: * transport_allocate_passthrough(). * iscsi_do_build_list(). Bart Van Assche. --
Greetings all, So I was able to FINALLY track this down to: -# CONFIG_SLUB_DEBUG is not set -# CONFIG_SLAB is not set -CONFIG_SLUB=y +CONFIG_SLAB=y in both your and Chris Weiss's configs that was causing the reproduceable general protection faults. I also disabled CONFIG_RELOCATABLE and crash dump because I was debugging using kdb in x86_64 VM on 2.6.24 with your config. I am pretty sure you can leave this (crash dump) in your config for testing. This can take a while to compile and take up alot of space, esp. with all of the kernel debug options enabled, which on 2.6.24, really amounts to alot of CPU time when building. Also with your original config, I was seeing some strange undefined module objects after Stage 2 Link with iscsi_target_mod with modpost with the SLUB the lockups (which are not random btw, and are tracked back to __kmalloc()).. Also, at module load time with the original config, there where some warning about symbol objects (I believe it was SCSI related, same as the ones with modpost). In any event, the dozen 1000 loop discovery test is now working fine (as well as IPoIB) with the above config change, and you should be ready to go for your testing. Tomo, Vlad, Andrew and Co: Do you have any ideas why this would be the case with LIO-Target..? Is anyone else seeing something similar to this with their target mode (mabye its all out of tree code..?) that is having an issue..? I am using Debian x86_64 and Bart and Chris are using Ubuntu x86_64 and we both have this problem with CONFIG_SLUB on >= 2.6.22 kernel.org kernels. Also, I will recompile some of my non x86 machines with the above enabled and see if I can reproduce.. Here the Bart's config again: What the LIO-SE Target module does is complex. :P Sorry for taking so long, I had to start tracking this down by CONFIG_ option with your I would be more than happy to point the release paths for iSCSI Target and LIO-SE to show they are not actual memory leaks (as I ...
This is also failing on CONFIG_SLUB on 2.6.24 ppc64. Since the rest of the system seems to work fine, my only guess it may be related to the fact that the module is being compiled out of tree. I took a quick glance at what kbuild was using for compiler and linker parameters, but nothing looked out of the ordinary. I will take a look with kdb and SLUB re-enabled on x86_64 and see if this helps shed any light on the issue. Is anyone else seeing an issue with CONFIG_SLUB..? Also, I wonder who else aside from Ubuntu is using this by default in their .config..? --nab --
I was able to track this down to a memory corruption issue in the in-band iSCSI discovery path. I just made the change, and the diff can be located at: http://groups.google.com/group/linux-iscsi-target-dev/browse_thread/thread/a70d4835c55... In any event, this is now fixed, and I will be generating some new builds for LIO-Target shortly for Debian, CentOS and Ubuntu. Espically for the Ubuntu folks, where this is going to be an issue with their default kernel config. A big thanks to Bart Van Assche for helping me locate the actual issue, and giving me a clue with slub_debug=FZPU. Thanks again, --nab --
Hey Jeff, I put a whitepaper on the LIO cluster recently about this topic.. It is from a few years ago but the datapoints are very relevant. http://linux-iscsi.org/builds/user/nab/Inter.vs.OuterNexus.Multiplexing.pdf The key advantage to MC/S and ERL=2 has always been that they are completely OS independent. They are designed to work together and actually benefit from one another. They are also are protocol independent between Traditional iSCSI and iSER. --nab PS: A great thanks for my former colleague Edward Cheng for putting this --
Actually, the iSER protocol wire protocol itself is quite simple,
because it builds on iSCSI and IPS fundamentals, and because traditional
iSCSI's recovery logic for CRC failures (and hence alot of
acknowledgement sequence PDUs that go missing, etc) and the RDMA Capable
Protocol (RCaP).
The logic that iSER collectively disables is known as within-connection
and within-command recovery (negotiated as ErrorRecoveryLevel=1 on the
wire), RFC-5046 requires that the iSCSI layer that iSER is being enabled
to disable CRC32C checksums and any associated timeouts for ERL=1.
Also, have a look at Appendix A. in the iSER spec.
A.1. iWARP Message Format for iSER Hello Message ...............73
A.2. iWARP Message Format for iSER HelloReply Message ..........74
A.3. iWARP Message Format for SCSI Read Command PDU ............75
A.4. iWARP Message Format for SCSI Read Data ...................76
A.5. iWARP Message Format for SCSI Write Command PDU ...........77
A.6. iWARP Message Format for RDMA Read Request ................78
A.7. iWARP Message Format for Solicited SCSI Write Data ........79
A.8. iWARP Message Format for SCSI Response PDU ................80
This is about as 1/2 as many traditional iSCSI PDUs, that iSER
encapulates.
--nab
--
this should be: .. and instead the RDMA Capacle Protocol (RCaP) provides the 32-bit or greater data integrity. --nab --
And a variant that doesn't do ATA or IP: http://www.fcoe.com/ --
however, and interestingly enough, the open-fcoe software target depends on scst (for now anyway) --
On Mon, 4 Feb 2008 20:07:01 -0600 STGT also supports software FCoE target driver though it's still experimental stuff. http://www.mail-archive.com/linux-scsi@vger.kernel.org/msg12705.html It works in user space like STGT's iSCSI (and iSER) target driver (i.e. no kernel/user space interaction). --
But ATAoE is boring because it's not IP. Which means no routing, firewalls, tunnels, congestion control, etc. NBD and iSCSI (for all its hideous growths) can take advantage of these things. -- Mathematics is the supreme nostalgia of our time. --
.. and all this could equally well be done by a simple bridging protocol (completely independently of any AoE code). The thing is, iSCSI does things at the wrong level. It *forces* people to use the complex protocols, when it's a known that a lot of people don't want it. Which is why these AoE and FCoE things keep popping up. It's easy to bridge ethernet and add a new layer on top of AoE if you need it. In comparison, it's *impossible* to remove an unnecessary layer from iSCSI. This is why "simple and low-level is good". It's always possible to build on top of low-level protocols, while it's generally never possible to simplify overly complex ones. Linus --
Never discount "easy" and "just works", which is what IP (and TCP) gives you... Sure you can use a bridging protocol and all that jazz, but I wager, to a network admin yet-another-IP-application is easier to evaluate, deploy and manage on existing networks. Jeff --
I frankly think NBD is at a pretty comfortable level. It's internally very simple (and hardware-agnostic). And moderately easy to do in silicon. But I'm not going to defend iSCSI. I worked on the first implementation (what became the Cisco iSCSI driver) and I have no love for iSCSI at all. It should have been (and started out as) a nearly trivial encapsulation of SCSI over TCP much like ATA over Ethernet but quickly lost the plot when committees got ahold of it. -- Mathematics is the supreme nostalgia of our time. --
Hi all, And sorry for intrusion, i am not a developer but i work everyday with iscsi and i found it fantastic. Altough Aoe, Fcoe and so on could be better, we have to look in real world implementations what is needed *now*, and if we look at vmware world, virtual iron, microsoft clustering etc, the answer is iSCSI. And now, SCST is the best open-source iSCSI target. So, from an end-user point of view, what are the really problems to not integrate scst in the mainstream kernel? Just my two cent, -- So long and thank for all the fish -- #Matteo Tescione --
The fact that your last statement is conjecture. It's definitely untrue for non-IB networks, and the jury is still out on IB networks. James --
On Tue, 05 Feb 2008 05:43:10 +0100 Currently, the best open-source iSCSI target implemenation in Linux is Nicholas's LIO, I guess. --
This is exactly how iSCSI-SCST (iSCSI target driver for SCST) is implemented, credits to IET and Ardis target developers. Vlad --
Sorry for the late response (but better late than never). One may claim that STGT should have lower performance than SCST because its data path is from userspace. However, your results show that for non-IB transports, they both show the same numbers. Furthermore, with IB there shouldn't be any additional difference between the 2 targets because data transfer from userspace is as efficient as data transfer from kernel space. The only explanation that I see is that fine tuning for iSCSI & iSER is required. As was already mentioned in this thread, with SDR you can get ~900 MB/sec with iSER (on STGT). Erez --
My most recent measurements also show that one can get 900 MB/s with
STGT + iSER on an SDR IB network, but only for very large block sizes
(>= 100 MB). A quote from Linus Torvalds is relevant here (February 5,
2008):
Block transfer sizes over about 64kB are totally irrelevant for
99% of all people.
Please read my e-mail (posted earlier today) with a comparison for 4
KB - 64 KB block transfer sizes between SCST and STGT.
Bart Van Assche.
--
And now consider if one target has zero-copy cached I/O. How much that --
