login
Search
Hello James and others,
During the open panel, my question was really specific -
Q) What is the future of a SCSI-target subsystem in linux. Which
target engine/subsystem can we expect?
Your answer) There is place for only 1 target-subsystem in the Linux scsi stack and in the LSF summit the decision was taken to merge LIO. Has that
decision changed since the summit?
3) above is something that I emailed Vlad and the scst community based on our offline conversation after the open panel. If SCST really has licensing issues then I will personally stop using SCST. Since Vlad hasn't
expressed any concerns on the above and neither have you commented on it, is it safe to assume that the licensing requirement is a non-issue?
Chetan Loke
--
The decision hasn't been taken to merge LIO, but based on what happened at the summit, I think it's the most viable candidate and will likely be No. James --
On Wed, Aug 18, 2010 at 5:11 PM, James Bottomley (resending as plain text) No matter which kernel-based storage target subsystem is merged, a migration path should be provided for SCST users such that these can continue using their SCST target drivers without too much trouble. Otherwise you'll upset a large number of SCST users. Note: an SCST target driver is the code that implements a storage protocol. See also http://scst.sourceforge.net/scst_pg.html for a overview diagram. Bart. --
On Wed, Aug 18, 2010 at 11:11 AM, James Bottomley During the open panel, facebook guys and others were tooting that start-ups thrive because they can hack linux. Well there are quite a few start-ups that use scst too for creating target appliances. Has anyone even bothered to glance the scst mailing list to see if that community is dead or alive? I for one use scst to create synthetic work-loads and test 200+ VM nodes in an ESX cluster. Anyone who has worked on a SAN OS will appreciate the simplicity of SCST. And if folks still can't understand the SCST code(after reading the README) then they are still welcome to send an email on SCST. Would you like to make your FC stack go faster, well please drop us an email on SCST and we will try our best to further optimize the FC driver. I know folks who don't understand simple DMA bus traces, FC wire traces and yet they have the power to influence decisions. James you are an expert but not everyone is. This is not a venting session but even folks who are new to target architecture find it easy --cloke --
But that's not really relevant, is it? I would expect that whatever I do, even keeping both out of tree, the communities around the solutions would still exist and be vibrant, since they're out of tree now, nothing will have changed. James --
The problem that people do believe that the merged code is the best code, so the being merged is an immediate HUGE advertisement. So, you can't say if a code is inside the mainline tree or not isn't relevant. This is the source of the questions from the SCST community. SCST is at the moment the best code. It's several years ahead any competitor. SCST has more functionality, SCST has more users, SCST has bigger community, SCST has more testing and, hence, stability, SCST has more support from storage vendors, etc. But for some reasons all those suddenly ignored and a raw, pursuing SCST code preferred instead. So, we want to know those reasons. SCST wasn't even really considered. Isn't Linux anymore a place where the best code wins? Frankly, it looks like the decision was done under closed doors based on rather political reasons, like personal connections... Vlad --
The problem that people do believe that the merged code is the best code, so the being merged is an immediate HUGE advertisement. So, you can't say if a code is inside the mainline tree or not isn't relevant. This is the source of the questions from the SCST community. SCST is at the moment the best code. It's several years ahead any competitor. SCST has more functionality, SCST has more users, SCST has bigger community, SCST has more testing and, hence, stability, SCST has more support from storage vendors, etc. But for some reasons all those suddenly ignored and a raw, pursuing SCST code preferred instead. So, we want to know those reasons. SCST wasn't even really considered. Isn't Linux anymore a place where the best code wins? Frankly, it looks like the decision was done under closed doors based on rather political reasons, like personal connections... Vlad [Jack]Vote for SCST as a user and target driver developer based on SCST. SCST really do good job. --
+1 from a happy SCST end-user. Regards, Ruben Laban Senior Network and Systems Administrator ISM eCompany --
You forgot to mention one small thing. You are going to (re)use current STGT interface for user space backend, which in 5 years of being mainline has not gained any noticeable interest, because it is fundamentally slow. It needs a complete redesign to become fast. In contrast, interface provided by scst_user has just 3% overhead comparing with fully in-kernel backend and allows to build storage capable of handling 1,500,000 IOPSes (Kaminario). Vlad --
First, 'build upon' and blind '(re)use' are different approaches. Building on is an important requirement for working with any existing mainline code regardless of how much much attention the code itself may require. Initially re-using pieces of the mainline code is acceptable to get a prototype up and running, and since we don't have many users for this piece of STGT, it will easier to make larger changes (eg: move beyond simple re-use) without breaking a large legacy base. This is what I actually prefer moving forward, as it gives us more Secondly, not being the original author of drivers/scsi/scsi_tgt_*, I am happy to do the work and submit the necessary patches to achieve the desired goal. Also, considering now we have the TCM v4 HBA/DEV abstraction with a fabric independent control path in target_core_fabric_configfs.c, there suddenly new interest to build upon tomo's and mnc'c original STGT work in drivers/scsi/scsi_tgt_*.c Using struct scsi_cmnd allocation from a specially allocated struct request_queue still my preferred method for doing this, unless tomo and mnc state otherwise. Also if we need to change the request_queue from being per struct Scsi_Host to struct scsi_device that is one thing that will be fine. If we need to make more drastic changes to drivers/scsi/scsi_tgt_if.c that is also fine. However saying that it needs a 'complete redesign', especially without ever consulting or offering to creat patches with STGT guys is not how Great! Please understand that you are more than welcome to create your own TCM subsystem plugin for your SCST kernel <-> user passthrough interface so it can take advantage of all the drivers/target/ code has to offer. Also, now that target_core_iblock.c, target_core_pscsi.c, target_core_file.c, and target_core_stgt.c are seperate standalone LKMs loaded on-demand by TCM/ConfigFS, creating a new TCM struct se_subsystem_api module will be even easier for you now. I would even be happy to send a functioning struct ...
That would be a bad design, because it would couple together fundamentally separate things: target and initiator subsystems (server and client, eg apache and firefox, sendmail and mutt, etc.). It would make the code harder to maintain and more fragile for modifications. On the user level it would lead to the need to have target mode core module always loaded. It is already so with STGT if you use FC or SRP. With them the only way to not have scsi_tgt loaded is to remove STGT on the Well, it isn't my habit to bother people asking something about what I can find an answer myself. I have spent sufficient amount of time hacking Linux kernel (>10 years, from which 8 in the target mode), so I can read others C code as easy as a book. I've already described which flaws I see in the STGT design and nobody objected me (one of the objections I repeated above). You can find my messages in the list archive. Isn't answering on exact critics a way how a cooperative development is supposed to be done? If somebody comes with a better solution for an existing problem is he supposed be ignored? Is Well, please understand that you are more than welcome to stop reinventing a wheel and instead just have the functionality and the advantages you referring already long ago implemented in SCST and being used to build (using your expression) records breaking storage products. You are also more than welcome to add the only extra functionality LIO has over SCST, ALUA, into SCST. Your patches are always welcome. Vlad --
That's not exactly what the results of a speed comparison one of your people did said, now is it? The results were actually not much difference on line speeds up to GigE. Interface re-use (or at least ABI compatibility) is the whole point, For a replacement, first we get the current userspace code working as is, then you can propose modifications to make it go faster. James --
On Sat, Aug 21, 2010 at 10:43 PM, James Bottomley I assume that you are referring to this message: http://lkml.org/lkml/2008/1/29/387 ? You might have missed the reply that was posted by Roland Dreier, a highly regarded kernel maintainer and InfiniBand expert (http://lkml.org/lkml/2008/1/29/402): "Maybe I'm all wet, but I think iSER vs. SRP should be roughly comparable. The exact formatting of various messages etc. is different but the data path using RDMA is pretty much identical. So the big difference between STGT iSER and SCST SRP hints at some big difference in the efficiency of the two implementations." Furthermore, I would like to clarify that Vlad hasn't asked me to start working on the SCST project but that I selected the SCST project myself after an extensive stability and performance comparison of four existing open source storage target projects. Bart. --
So the phrase "up to GigE" was deliberately in the above to exclude the disputed infiniband results. I'm not really interested in re-opening the arguments over how to interpret those results. The fact that SCST and STGT were on par up to 1GbE is enough to refute the contention that STGT is "fundamentally slow". --
We haven't tested this in at least 6 months, but we did test iSCSI over 10GbE using SCST and STGT. This was one of the reasons we wound up going with SCST. For the past several years, our performance on SCST was much higher than on STGT. If it helps, we can redo these tests with a modern kernel (2.6.35.x) and same backend/frontend. We've been switching most of our IO performance testing to Jens Axboe's excellent fio tool. I'd be happy to share our testing decks with anyone (and collaborate on generating a good useful set for general use) This said, I have to add in my support to SCST and its developers. Vlad, Bart, and the rest of the community have been very helpful to us. We have been a consumer rather than a developer to date, so we have less of a dog in this fight than others. We have nothing against LIO or STGT. We will try them and see if they meet our needs. SRP is a hard requirement, and it exists and works great in SCST. So for us, a solution which gives us the best of all worlds would be one where we don't have to give up what works well for us, and gets us new capabilities/features. This is more of a wish-list ... we have to keep using what works well for us. Our interest in STGT is that we would like a working iSER. Vlad has suggested a method that we will explore a little later on (next month). The LIO folks do look like they have some interesting capabilities beyond this, so we will look. But without SRP, and a fast iSCSI, their really isn't much choice for us in the near term. Regards, Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: landman@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/jackrabbit phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 --
On Mon, Aug 23, 2010 at 3:47 PM, Joe Landman (resending as plain text) There is an important design difference between SCST and LIO: SCST by defaults creates multiple threads to process the I/O operations for a storage target, while LIO only creates a single thread per storage target. This makes SCST perform measurably faster. Bart. --
Forget that. You could have discussed this if there were code reviews or other mainline inclusion emails from James B. From what I have heard, the decision was taken around 8-9 months back. Would anyone like to either comment/validate/refute this please? If not then I would kindly request these guys to stop taking us for a test drive. And also I'm not sure when was the last time James B. bench-marked our scsi-stack. Even if I ACK in the xmit-path then I can't push more than 100K IOPs. But other folks have re-engineered our linux-scsi stack and from what I've heard they can push > 300K+ IOPs. So I would just ignore performance discussion because I don't think folks have done even simple lame experiments in the last 1 year. Or may be I'm completely wrong and so please enlighten me so that I can Chetan Loke --
I actually received 3+ off-post emails asking whether I was talking about initiator or target in the 100K IOPS case below and what did I mean by the ACKs. I was referring to the 'Initiator' side. ACKs == When scsi-ML down-calls the LLD via the queue-command, process the sgl's(if you like) and then trigger the scsi_done up-call path. Chetan Loke --
Uhm, Intel and Microsoft demonstrated over 1 million IOPS using software iSCSI and a single 10 Gbit Ethernet NIC (Intel 82599). How come there is such a huge difference? What are we lacking in Linux? --
I also have an impression that Linux I/O subsystem has some performance problems. For instance, in one recent SCST performance test only 8 Linux initiators with fio as a load generator were able to saturate a single SCST target with dual IB cards (SRP) on 4K AIO direct accesses over an SSD backend. This rawly means that any initiator took several times (8?) more processing time than the target. Hardware used for that target and initiators was the same. I can't see on this load why the initiators would need to do something more than the target. Well, I know we in SCST did an excellent work to maximize performance, but such a difference looks too much ;) Also it looks very suspicious why nobody even tried to match that Microsoft/Intel record, even Intel itself who closely works with Linux community in the storage area and could do it using the same hardware. Vlad --
You seem to be under the impression that "Intel" is some monolithic entity. Despite working with six different storage & performance groups within Intel, I have no idea what record you're referring to, nor what hardware it was accomplished with. Even if I did, I wouldn't know which group within Intel to contact to see if they still have the setup. Then I'd have to convince them that it's in their interest to try to replicate this on Linux. And I'd have to be prepared to sink a considerable quantity of my time into it ... which I don't have. -- Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step." --
It is Sorry if it looked like I was blaming you. I just was wondering why Intel developed Linux drivers for those network adapters and isn't interested to similarly demonstrate their performance on Linux. Vlad --
Ah, iSCSI. I don't work with that group. I'm a bit too busy with other projects to pursue a relationship with them right now :-) -- Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step." --
Uhm, that's MS(and it's closed tcp-chimney protocols and other offloads?). And I think we discussed in bits and pieces about this on scst already. Also, just because the driver is open sourced in linux may not necessarily mean that we know all the ASIC registers that we can bit-bang and squeeze every clock cycle out of the ASIC(just a I'm not a iscsi-guy. So I can't comment on how the data is moved from --
While I can't tell you where the bottlenecks are, I can share some performance numbers... 4 initiators can get >600K random 4KB IOPS off a single target... which is ~150% of what the Emulex/Intel/Microsoft results show using 8 targets at 4KB (their 1M IOPS was at 512 byte blocks, which is not a realistic test point) here: http://itbrandpulse.com/Documents/Test2010001%20-%20The%20Sun%20Rises%20on%20CNAs%20Te... The blog referenced earlier used 10 targets... and I'm not sure how many 10G ports per target. In general, my target seems capable of 65% the local small-block random write performance over IB, and 85% the local small-block random read performance. For large block performance, ~95% efficiency is easily achievable, read or write (i.e. 5.6GB/s over fabric, where 6GB/s is achievable on the drives locally at 1MB random blocks). These small-block efficiencies are achievable only when tested with multiple initiators. The single initiator is only capable of <150K 4KB IOPS... but gets full bandwidth w/ larger blocks. If I were to chose my problem, target or initiator bottleneck, I'd certainly rather have an initiator bottleneck rather than Microsoft's The numbers are suspicious for other reasons. "Random" is often used loosely (and the blog referenced earlier doesn't even claim "random"). If there is any merging/coalescing going on, then the "IOPS" are going to look vastly better. If I allow coalescing, I can easily get 4M 4KB IOPS, but can't honestly call those 4KB IOPS (even if the benchmark thinks it's doing 4KB I/O). They need to show that their advertised block size is maintained end-to-end. Chris --
Hmm, on the data you sent me only 8 initiators were capable to do so... From my, a storage developer's, POV it isn't about if this test is realistic or not. 512 bytes tests are good if you want to test how processing effective your I/O stack, because they produce the max possible CPU/memory/hardware interaction load. Since processing power isn't unlimited, in case if it is a bottleneck, N IOPS on 512b < N IOPS on 4K * 8 and system with more effective processing will have better numbers. Vlad --
I've been asked to share more details of the single SRP initiator
case, comparing Windows to Linux...
The configurations tested are represented by four digits separated by dashes:
- The number of initiators used in the test (always one in this case).
- The number of target ports used.
- The number of initiator ports used.
- the number of drives used.
SRP Upstream Initiator
1-1-1-1 1-1-1-2 1-2-2-2 1-1-1-4
1-2-2-4 1-1-1-8 1-2-2-8
Random Write 122880 141568 206592 144384 163840 141824 165376
30/70 R/W mix 72113 123136 144640 143616 163072 145920 163584
70/30 R/W mix 55938 91392 114176 135680 156160 145920 162304
Random Read 50688 78336 107008 121600 149760 143872 161536
SRP Windows Initiator
1-?-1-1 1-?-1-2 1-?-2-2 1-?-1-4 1-?-2-4
1-?-1-8 1-?-2-8
Random Write 57774 116738 114464 146972 202891 221819
30/70 R/W mix 49719 95697 97831 154328 181221 227786
70/30 R/W mix 45242 90694 89559 167341 176178 244661
Random Read 48016 94867 92984 178227 183631 257449
Note that the question marks are where I'm not sure how Windows is
using the second target port... in Linux, you select the target port
from the initiator, but there's no such option in Windows, so the
target port could be used in those cases. The 1-1-1-8 case is where I
tried to force it to use just one target port (by disabling the target
port), and Windows wouldn't do any I/O at all.
--
Well, James, why not 100MbE? If you want a comparison of target implementations you need a fast hardware with minimal latency. Otherwise, the difference between the implementations can drown in the overhead of the accompanying processing. 1GbE is a nearly 10 years ago interface. Or are we going to stay ten years behind progress? Thanks, Vlad --
I see now. You want ABI compatibility to keep the "contract" that no kernel changes can break applications binary compatibility for unlimited time. OK, we will write the compatibility module. It shouldn't take much time. But before we start, I'd like to clear 2 related questions: 1. How far the ABI compatibility "contract" goes? Are there cases, where it isn't so strong? I'm asking, because I can recall that open-iscsi at least once has broken ABI compatibility with user space tools. Was it an accidental (but not corrected) mistake or was it deliberate? If the latter, then, I guess, there must be some exceptions defining when ABI compatibility can be not followed. 2. Currently STGT in the kernel is just 2 files, scsi_tgt_if.c and scsi_tgt_lib.c (with headers), + ibmvstgt driver. The C files define the STGT interface in question. So, if we keep ABI compatibility with the new target engine, we would have to keep those 2 files included in the kernel, which would effectively mean that STGT would stay in the kernel. This would lead to the situation you are trying to avoid: 2 SCSI target infrastructures in the kernel. Would it be OK? Thanks for (eventually!) defining clear rules and removing confusions! Vlad --
ok now I'm confused, or maybe I'm not understanding ABI correctly, or maybe you guys are using it in a way that is inconsistent with popular convention. As a VMware user, I have experienced fully that the kernel ABI changes in various places with every release. VMwares drivers have to be constantly updated to match changes in kernel function parameters and even what functions are available. I've also experienced it with scsi cards, dsl modems, and other 3rd party drivers. It's the one big downside to developing for the Linux kernel, the ABI is /always/ changing. --
You're talking about in-kernel ABI, which is constantly in flux. James is talking about user <-> kernel ABI, which is not supposed to change in an incompatible way. -- Matthew Wilcox Intel Open Source Technology Centre "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step." --
You are thinking of the KABI. That changes per each release except if you buy a vendor product. Red Hat for example keeps an KABI symbol list where they guarantee that those parameters, structures ,etc will never change. John Masters wrote a nice paper about how they solved this: http://dup.et.redhat.com/presentations/DriverUpdateProgramTechnical.pdf I don't have experience with other vendors. In terms of ABI, think ioctl calls and its a parameters. They are suppose to --
Just to make sure people aren't getting more confused. What Red Hat calls the KABI (and SLES and Ubuntu do something similar) is the Kernel ABI which is presented to kernel modules. This is important for companies shipping out-of-tree and proprietary kernel modules. (And we'll ignore the questions about whether such proprietary kernel modules violate the GPL or not; contact your favorite lawyer for an opinion on that subject. It depends on the facts of the case and your When we talk about the ABI must be constant, this is the kernel interface presented to userspace programs, including statically linked userspace progams. So system calls, ioctl's, etc. This is what allows you to download or purchase a userspace program (including silly things like DB2, or Oracle Database, or Adobe Flash), and it will work even if you upgrade your kernel. - Ted --
I don't think it has to be complete. As long as the STGT people think This isn't really correct. The ABI is defined by the headers not the If you mean is the marketing solution of wedging two products into the kernel and calling it a single one going to fly, the answer is no. James --
Yes, but we on the target side would not be able to implement the ABI compatible interface without using library functions provided by those C files. Or, at least, it would be much harder.
I mean that if we keep those 2 files to ease our ABI compatibility effort, it would effectively mean that we would leave STGT merged. It isn't something we would create, it just would be so itself as a matter of fact. Ultimately, STGT is an user space engine. What it has in the kernel is the interface helper functions to interact with the in-kernel drivers. The simplest way to achieve the ABI compatibility is to make a backend module acting as an STGT in-target driver.
(Actually, I may not ask it, because this is the way how LIO seems[1] implemented that, which was approved on the LSF summit. I only want to make all pros and cons clear from the very beginning.)
Thanks,
Vlad
1. I wrote "seems", because currently LIO has the following code for STGT commands execution:
int stgt_do_task(se_task_t *task)
{
stgt_plugin_task_t *st = (stgt_plugin_task_t *) task->transport_req;
struct Scsi_Host *sh = task->se_dev->se_hba->hba_ptr;
struct scsi_cmnd *sc;
int tag = MSG_SIMPLE_TAG;
sc = scsi_host_get_command(sh, st->stgt_direction, GFP_KERNEL);
if (!sc) {
printk(KERN_ERR "Unable to allocate memory for struct"
" scsi_cmnd\n");
return PYX_TRANSPORT_LU_COMM_FAILURE;
}
memcpy(sc->cmnd, st->stgt_cdb, MAX_COMMAND_SIZE);
sc->sdb.length = task->task_size;
sc->sdb.table.sgl = task->task_sg;
sc->tag = tag;
BUG();
#warning FIXME: Get struct scsi_lun for scsi_tgt_queue_command()
#if 0
err = scsi_tgt_queue_command(sc, itn_id, (struct scsi_lun *)&cmd->lun,
cmd->tag);
if (err) {
printk(KERN_INFO "scsi_tgt_queue_command() failed for sc:"
" %p\n", sc);
scsi_host_put_command(sh, sc);
}
#endif
return PYX_TRANSPORT_SENT_TO_TRANSPORT;
}
which means that this pluging completely not functioning.
--
Vlad, As mentioned explictly earlier in this thread, my WIP code for the kernel level subsystem backstore using STGT kernel <-> user CDB passthrough logic in drivers/target/target_core_stgt.c is a item on my TODO list, but I will repeat, is NOT being tagged as a mainline .37 item. Tomo-san, mnc, and other storage folks have been briefed on the remaining issues that would be involved to get a prototype functioning with drivers/target/target_core_stgt.c, and rough idea of what it would take in existing mainline drivers/scsi/scsi_tgt_*.c code. With the current WIP code this will allow the userspace CDB -> LUN passthrough to function transparently with all TCM SPC-4 compliant logic as a standalone struct se_subsystem_api tcm_core_stgt.ko backstore. Best, --nab --
Hmm, I can't understand, if target_core_stgt.c is "NOT being tagged as a mainline .37 item", then the STGT ABI compatibility for being a drop in This is exactly what we are discussing. Thanks, Vlad --
Sorry, but I have no idea what you are trying to conjour up here.
To spell out (again) the TCM/LIO<->STGT compatibility stages that have
been persued pubically over the last months:
I) Create proper userspace tgt.git SG_IO and BSG passthrough into
TCM_Loop v4 using high-level multi-fabric WWPN emulation so that TCM
Core SPC-4 kernel emulation is exposed to STGT user fabrics, eg:
userspace fabric module -> kernel backstore passthrough. (DONE
for .37, and STGT passthrough + BSG backstore support merged into
tgt.git by Tomo-san)
II) Complete target_core_stgt.c TCM subsystem plugin for kernel -> user
CDB -> LUN passthrough building upon existing
drivers/scsi/scsi_tgt_*.c code. (WIP for .38, made available
Then I suggest you start working on a patch for drivers/scsi/scsi_tgt_*
in order to address what you believe are the preceived shortcomings of
the original design.
Until you can do that, and actually take an impartial look at the
existing drivers/scsi/scsi_tgt_*.c, it would be a bit difficult to
beleive you genuinely want to steer our current level of interaction
away from continued hand-waving noise into a real rational technical
discourse between yourself and the LIO/STGT development community.
Best,
--nab
--
That would mean that if LIO merged in .37: 1. It would be merged _without_ STGT ABI compatibility, i.e. one of the requirements for a new SCSI target infrastructure merge is violated. 2. It would _not_ be a drop in replacement for STGT, because STGT's drivers/scsi/scsi_tgt_*.c code would stay in the kernel. Those would effectively mean that another requirement for a new SCSI target infrastructure merge would be violated: there would be 2 SCSI target infrastructures in the kernel and any STGT in-kernel driver (for instance, built as an outside module) would continue working. My understanding of "drop in replacement" is "one out, another in at once". Please tell me where I'm wrong? I would appreciate if you be precise and answer the above 2 my concerns. There is no point to again repeat what My look is completely impartial. With all my respect, I'm just quite ahead of you and can see what you are unable (or don't want?) to see. I did what you are currently doing almost 4 years ago. I have already written all the necessary code and addressed all _rose on practice_ concerns. But all my attempts to explain my view are just blindly ignored without any considerations, so I have no idea how more I can explain it. Thanks, Vlad --
Sorry, but you are completely wrong here. This has nothing to do with a question of STGT kernel 'ABI compatibility' (because there is only one mainline user!), but has everything to do with being able to expose TCM kernel level SPC-4 emulation, and make this logic available to existing userspace fabrics in tgt.git. Again, we are talking about userspace STGT fabric module <-> TCM kernel backstore support for .37, which has already been merged by into tgt.git, and being used by other STGT users Sorry, but this is just more generic handwaving on your part. What constitutes a 'drop in replacement' for STGT is a decision that was to be made by the STGT developers, not by you. target_core_stgt.c is an extraordinarly transparent method of bringing drivers/scsi/scsi_tgt_* logic into the TCM Core HBA/DEV design, and allows us to build upon and It is obvious to even an casual observer from watching the TCM/LIO patch series that have been flying across the linux-scsi wire the last 24 months that the major features (including PR and ALUA, and new fabric module drivers) have been developed individual feature bit by feature bit using a distributed git workflow in a bisectable manner. Each series was produced in such a manner that each patch could be reviewed individually by those interested parties. This is the big difference between our respective development processes. This is the case not only for SCSI feature bits that TCM/LIO and SCST share, but for features in TCM/LIO v4 that are unique and not available in any other target implemention, anywhere. And yes, I am most certainly talking about code beyond just the SCSI level fabric emulation Then I suggest you learn a better way of communicating your ideas if you really want to work with members of the LIO/STGT development communities. First, I suggest you start explaining your ideas with actual kernel code that is 1) human readable and 2) presented in such a manner that makes it accessable for others with skills possibly ...
You are proving that I'm actually right ;). In the beginning of this thread I offered a transition path from STGT to SCST and James rejected it, because it didn't confirm a requirement that a new SCSI target infrastructure must be STGT ABI compatible. But latter James left this decision up to the STGT developers and now you are confirming that they don't see any ABI compatibility issues, so my transition path fully I'm not deciding anything, I'm only analyzing and seeing a contradiction: 1. James wants only one SCSI target infrastructure in the kernel. 2. If drivers/scsi/scsi_tgt_*.c left after the merge of the new SCSI target infrastructure, it would mean that STGT left as well => 2 SCSI target infrastructure in the kernel. For me it doesn't matter. I just want clear rules, hence asking for That's a nice, but quite meaningless LIO advertisement. SCST is using the same bisectable, distributed and reviewable workflow. If we had a kernel.org git account, we would use it as well, but so far we are happy with SF.net hosting. BTW, how was you able to get the git account without a single patch merged in the kernel? You must have good I have sent patches twice, the second time few months ago. Weren't they human readable and accessible for kernel developers who are experts in dealing with sent by e-mail patches? Thanks, Vlad --
Again, I still have no idea what you are trying to conjour up. The passthrough for STGT compatibility with TCM_Loop via SG_IO and BSG has already been merged by Tomo-san into tgt.git. Futhermore, we are using the same TCM_Loop high level multi-fabric WWPN emulation together with new (and old) QEMU HBA emulation into KVM guest using SG_IO and BSG as Seriously, arguing over the use of people's language from weeks ago once the decision has already been made to move forward is really of little Actually, that is incorrect. Your project uses a centralized development model, which has it's obvious limitiations in terms of speed, flexability, and community scale. But really, don't take my word for it, you can hear it for yourself directly from the horse's mouth here: http://www.youtube.com/watch?v=4XpnKHJAok8 I also very strongly suggest you find a transcript of this talk so you can really understand what Linus means here wrt to a distributed Please don't turn this into a kernel.org vs. SF.net thing.. There are Well, amougst the biggest advantages is actually having a sane ConfigFS interface that can represent the parent / child relationship between data structures across multiple LKMs. Not only does this give us a proper representation of TCM and fabric data structures that can be accesses directly by an advanced user in /sys/kernel/config/target/, it also provides an ideal foundation to build 1) a userspace library in intrepted languages and 2) create high level applications using said userspace libraries that allow compatiblity to be contained in the lib below. But wrt to actual kernel code (because this is a kernel list), the shortlog for the RFC posting below nicely sums up what TCM v4 is bringing to the table: I think a proper kernel subsystem maintainer workflow has to do alot more to do these days than just sending patches over email, than it did say 10 years ago. Like it or not, git is the language of the mainline kernel development process, and ...
The patches were good, but don't mislead people about that. For sake of clearness, what you called "the passthrough for STGT compatibility with TCM_Loop via SG_IO and BSG" has nothing with LIO/TCM_Loop. What merged fixed problems STGT had in the pass-through implementation. It at the same time fixed problems with SCST's scst_local, so similarly can be called "the passthrough for STGT compatibility with scst_local via SG_IO and BSG". The same is for the second half of the above. It's for scst_local in the Does it look I'm arguing? Again, I'm analyzing and asking for clarification in the contradictions I see. I don't like when such Well, Nicholas, are you really understanding what you are writing? We in our projects have fully the same distributed (or centralized, if you like it) development model. Have you noticed how many developers SCST project has? We have our responsibility areas (target drivers, scstadmin, etc.) and commit in them. The way how we get updates for the rest of the kernel doesn't matter. Git is better for such huge projects as the kernel, but for our relatively small and centralized by nature due to small size (sub)projects it doesn't matter and won't bring any value. Actually, to have all in one place without the rest of the kernel is more convenient, because grep over few MBs is a lot faster than over the SF.net for a long time has been offering git hosting, but we don't see reasons to move to it. SVN has all we need. We are not (yet) so big to need git. But, we can move to git at any moment as soon as it is needed, for I have no doubts in benefits to TCM. But I have asked about benefits _for an average user_. STGT already can do what you listed above. Also what advantages and additional value it has over what SCST So, do you believe that as soon as we migrate to git all the obstacles for the SCST merge would be immediately removed? Thanks, Vlad --
Uh, I don't think TCM_Loop and scst_local are exactly comparable at this point. TCM_Loop supports high level multi-fabric WWPN emulation that allows it to work transparently with inter-nexus SPC-4 PR operation. Therefore, using TCM_Loop, STGT userspace fabrics now support both PR and ALUA emulation from TCM_Loop LUNs. This is done at struct config_group_ops->make_group() time to report $PROTO_IDENT and the necessary WWPN info to TCM Core logic. TCM_Loop also uses the fabric indepent configfs handlers for pretty much all of it's control plane I really have no idea what you are talking about 'behind closed doors'.. Have you not been watching the amount of TCM/LIO patches on the I find it hard to beleive you are actually going to agrue against a git workflow for a target mode subsystem maintainer, well considering that git was made by a linux kernel maintainer (Linus) for distributed linux kernel development (everybody else)..? Again, I suggest you watch the Russian translation of that talk on youtube and really understand what he means, aside from the insults to Again, from above how it benefits users: 1) a userspace library in intrepted languages and 2) create high level applications using said userspace libraries that allow compatiblity to be contained in the lib below. 2) create high level applications using said userspace libraries that Uhhh, I don't think so. Aside from your obvious project workflow issues, the fact that SCST completly lacks a proper user-space driven representation of parent / child relationships between kernel level data structures in a fabric independent manner, while still allow for fabric dependent groups and attributes is a most definately show-stopper for me. Best, --nab --
This is complete BS. Are we going to judge value of a project based on what kind SCM they chose to use? I guess we should ban Greg KH from kernel development and rip out USB and driver core layers from the kernel because he has the audacity to use quilt for his trees. -- Dmitry --
I think the difference between what Linus says and what he actually means in the above video could be easily misinterpreted, well especially for some non-english speaking folks. But I am certainly glad to see that a russian translation is also available for this google git talk for those who really want try to understand his reasons (and technical requirements) for what he is saying. So as to the specifics about why git is really the only right SCM choice for mainline target mode maintainership, it really all comes down to workflow. Does your SCM allow other people to easily and without-pain track your upstream subsystem tree changes and 'rebase' as necessary for their patch series you make improvements..? If we are talking about say, a single standalone driver being developed against mainline, then sure using a SCM like CVS or SVN is perfectly acceptable when you push to someone upstream like gregkh or akpm via email patch attachments. However, if we are talking about a subsystem maintainer workflow that requires many different people to track your subsystem tree for their own drivers and new feature bits, not being able to keep local branches for these level developers makes their life excruciatingly painful and unpleasent as they attempt to pull new upstream changes, especially when the speed of new upstream development is moving quickly. This rule applys even more when said subsystem has a greater scope within the source tree in question. Anyways, if we are going to compare SCM distributed vs. centralized workflow in terms of kernel projects, lets please at least compare apples to apples here. Best, --nab --
On Sun, Sep 5, 2010 at 5:50 PM, Nicholas A. Bellinger I've been trying to keep my 2 cents out of this, as I am merely an SCST user. Both of you have valid criticisms; the choice of SCM is not one of them. It is nit-picking at best, especially when SCST could switch to git easily if they so desired. Seven years ago, would you be pushing BitKeeper? Kind Regards, Mark Deneen --
Hi Mark, I will always be advocating using the best tool for the job in any given situation. So absoulutely, I would have picked bitkeeper over tarballs any day of the week 7 years ago, or over SVN if it had existed back then. But again, I think it's an important point that git is a tool that was made explictly for the linux kernel workflow. Why would a new subsystem maintainer is participates in the kernel workflow ever use anything besides git at this point..? And sorry, but considering the obvious advantages in terms of workflow speed and flexibility that git brings to the table for a subsystem maintainer, calling the choise of SCM a nit-pick item demonstrates a level certain level of inexperience wrt to mainline kernel workflow. Which is perfectly OK, but if you really want to understand the issues at hand in a distributed vs. centrailized SCM model, I strongly suggest you watch Linus's talk as well. Best, --nab --
I can't say that I agree with this. SVN existed, along with many Look, I'm not saying that I dislike git. I use it as my SCM here. However, git was in its infancy (or not even around) when SCST was started. It's not like they had a proprietary vendor go cold turkey I'm still calling it a nit-pick. Vlad could switch to git in a short amount of time if he felt so compelled. This is like saying that the quality of a car is based on the style of garage it is parked in. Kind Regards, Mark Deneen --
Bitkeeper taught Linus by his own admission that there was actually a reason to using a SCM for the kernel to begin with, and helped drive some early git design princables which he also briefly mentioned in the google git talk. So I hardly consider this a mistake looking at it from a historical I am really sorry to hear about SCST's bad timing wrt to the evolution of git, but I hardly see this as an acceptable excuse for poor mainline Well, if we are going to start talking about car analogies, then I have one for you.. 8-) Using a centralized SCM for kernel subsystem workflow in the year 2010 in akin to trying to make a modification to a 18,000 RPM capable engine in a Ferrari F1 (eg: Linux Kernel), tuned to run at the *highest* levels of international competition (eg: LKML). But instead of using the tools (git) that where explictely designed the F1 engine by it's creator (eg: Linus aka Enzo Ferrari), you end trying to adjust your F1 engine's killowatt per litre displacement output using a broken FM tuner knob and rusty spare tire jack from a 79' Ford Pinto. Best, --nab --
Will you please stop being condescending? Yes, it is nit-picking. Many subsystems and features are being developed using patch series stored not in git but somewhere else. Have you noticed that akpm uses his own patch utils because git is not the best tool _for him_. Git is perfect for Linus's and DaveM workflows (as they in turn do pulls), whereas maintainers that prefer patch-based submissions may or may not use git, depending on their preferences. The choice between 2 implementations should be based purely on design and code quality (with maintainer being reasonable and flexible enough for additional points). -- Dmitry --
Will you please piss off with your insinuations that we do not Understand English? While it may not be our first language it does not Haven't you noticed yet that not every maintainer uses git? USB, driver core, tty, staging and i2c subsystems are based on quilt queues. Media/DVB main tree is in mercurial. AOE is quilt. And I am pretty sure there are others, I just don't care about them. And if you prefer to use No, we should not be comparing SCMs at all here but rather 2 competing implementations based on quality of the code. You tried to bring SMC angle in and I am saying that it is BS. -- Dmitry --
Again, without getting into another pointless flamewar, I think the main point here is that a open source project using a distributed workflow (like git) has major advantages when it comes to working with a larger group of developers than a centralized model (like SVN) does. Because being a subsystem maintainer typically involves this type of complex workflow involving lots of different parties, git is a tool that was created (originally) expressely for a kernel workflow, and for those types of people it works really, really well. So, please understand that code and project workflow is only one of the reasons why TCM/LIO v4 was selected over SCST. I invite you to take a closer look at the RFC Code that has been posted last week if you want to get into the nitty-gritty techinical details, which this thread has thus far been avoiding. Best, --nab --
You may be surprised but I am aware of subsystem maintainer workflow. I also am aware that there is not a single workflow and that it varies by maintainer. For some git is indeed the best tool but others (and I named a few) might prefer something different. Once again: should we declare all subsystems that do not use git as Yes, I agree that it would be much more productive if you concentrated on technical details when answering Vlad's questions rather than branching into immaterial argument over SCM choice. -- Dmitry --
Oh, for god's sake children. Why does every LIO vs SCST discussion turn into a pointless flameware over stuff no-one really cares about? If none of you has anything substantive to say: don't say it. Since patches into SCSI go over the mailing list for review and integration (and running checkpatch.pl on ... this would be a hint), I It isn't yet ... your code still has to be reviewed properly. My preferred reviewer is currently honing his skills on a diet of raw beef in Argentina, but hopefully he'll get around to it shortly. James --
On Mon, Sep 6, 2010 at 12:39 PM, James Bottomley The SCST developers and the SCST users will definitely appreciate it if review efforts will be spread evenly over both projects. Bart. --
SCST isn't at the starting gate yet, so there's nothing currently to review. James --
Hmm, there are patches for review for several months (http://lkml.org/lkml/2010/4/13/146). Nothing major changed since that time. But we are going to send patches for the latest code this week anyway. Vlad --
James, sorry, but you can't blame us. I keep asking for clear rules and don't receive much. So, there are speculations and pseudo-rules, which sometimes go to the absurd, as in this SVN vs Git case. No surprise then, that people have risen against this absurd (thanks a lot to them for support!). Frankly, in all the situation around Linux SCSI targets I for quite a long time feeling myself as a hero of a Kafka novel. Supposed to be goals are to have the best code doing its job the best, but on practice nobody cares about technical arguments and figuring out which subsystem is the best. Instead, everything lives it own incomprehensible life, where doesn't matter what you are doing, all already long ago decided behind your back and you will never be told why. All accurate statements not heard or, at best, called "handwaving", but dirty public opinion manipulations based on half- and less-than-half- truth have very warn Great to hear that! Thanks! Vlad --
Sorry Vlad, but this is simply not the truth. You have had ample time to comment on the hundreds of TCM/LIO patches posted to linux-scsi and lkml over the last years, but you have chosen never to comment on a *single* one then, or even on a single one now of the dozens that have been posted in the last 3 weeks while this thread has been lumbering forward.. So at this point, I will once again to refrain from any non technical interaction with yourself. If you have geninue concerns about any of the TCM/LIO v4 code, then I suggest that you and your devels make them known from within threads containing [PATCH] and [RFC] tags, because I will not be bothering with anything that does not contain comments on creating new or improving existing design and code. Best, --nab --
I haven't seen any techinical discussion from you either. all I see is how your favorite has been blessed as being the next thing to be included, even though it still needs work and Vlad's can't even be looked at. that's hardly a technical discussion. Vlad seems to be trying to document features of the different options, naturally he knows his option better than anyone else's. He offered to correct his listing with any information provided by anyone else, and instead of corrections to the feature lists, we just got accusations of handwaving and a reiteration that your favorite was selected in some meeting to be merged. I would like to point out that over the years there have been many things that were expected to be merged that didn't get merged. Don't take any statement in any meeting to be a final decision. It doesn't matter how promising yourcode looked at that point in time, until it's merged you may find that it doesn't get merged. And even after it gets merged, if someone else has better code, their code may replace yours if the other code is better. As a user, I would sure like to see more information about the two major choices and a lot less "this is what was picked at an invitation-only meeting where one option wasn't represented" David Lang --
I think this is somewhat backwards... Vlad appears to be asserting that SCST is more feature-complete that LIO at this point. It also seems that LIO is somewhat younger than SCST. So at this point it might be interesting to see: 1. What are the shortcomings of SCST design compared to LIO and why LIO developers chose to come with their own solution instead of collaborating with SCST folks? 2. What features are missing from SCST that are currently available in LIO? Once this is sorted out and [most] everyone agrees that LIO is indeed technically superior (even if maybe not as mature yet) solution, then it would make sense to request SCST developers to go to file/line depth of the review. Thanks. -- Dmitry --
On Mon, Sep 6, 2010 at 8:44 PM, Dmitry Torokhov I would also appreciate an overview or a block diagram of how things work in LIO. I hope that's not too much to ask for? That way we can compare/contrast how things work from 10,000 feet level. I for one don't want to look at a single patch and comment - i) Oh, change this variable here because it doesn't follow linux coding style. ii) You dropped a lock in queue-cmd? Good. qla's driver has been doing it for sometime, no? So you could have just looked at that LLDD. Chetan Loke --
I agree with you. Not only the code of the two projects should be reviewed but their respective designs should be reviewed too. Bart. --
On Tue, Sep 7, 2010 at 2:44 AM, Dmitry Torokhov You seem to have missed the start of this thread. The design of SCST is significantly more advanced than that of LIO, and it has already been explained in this thread why (http://www.spinics.net/lists/linux-scsi/msg45856.html). Bart. --
The question was directed to LIO folks as they appear to disagree with this statement. -- Dmitry --
Those are exactly the questions trying to hear answers on which I'm hitting the wall in past time. Thanks! Vlad --
Actually, http://scst.sourceforge.net/comparison.html is the result of a very careful and comprehensive review of your code, most likely, the best it has ever had. Until high level questions are answered (Dmitry perfectly summarized them), there's no point to go into low level details reviewing particular patches. Vlad --
I actually use git to manage my quilt patch tree so that others can work with me. Don't confuse a scm with a patch management tool, they are two vastly different things and have no relation to each other here. And this sub-topic has NOTHING to do with the main issue here, so I don't know why you are arguing over it. Please take it elsewhere. thanks, greg k-h --
On Thu, Sep 2, 2010 at 4:25 PM, Nicholas A. Bellinger Please don't make me go public. I'm trying to keep my mouth shut purely out of NDA. Inspite of that I've mentioned it publicly in my last emails that the decision was made ~9 months ago? Someone told me 'LIO will be merged eventually and NOT scst'. This ain't open-source development as we know it. Also there are exactly 'ZERO' users/developers yelling 'NOT' to include scst. So please stop your PR about LIO. Since no target subsystems were ever chosen, I would think that LIO's list should have some meat, no?And by now, the whole community knows that you have near to zero meaningful/technical conversations on LIO's mailing list. So I'm not sure when you talk about your git workflow and your patches, Nothing needs to bisected. Just go to scst's svn repo and see the checkin-flow over there if you want to satisfy your thirst. There are no rules defined whatsoever that says that the patches need to follow git-workflow. Since you are not an official kernel subsystem maintainer no need to send you patches to get a buy-in. svn is more than capable for what it is supposed to do. So no need to An average user doesn't implement storage blobs. So I don't want to worry about what they would think.This is your useless LIO PR and not Show stopper for you? Sorry but you seem to be self-proclaimed judge/maintainer pushing/speaking your vendor's/customer's agenda. Ever noticed how screwed up the fc representations were? Take NPIV for instance. No one laid down rules at that time? And just because your so called vendor's/customer's may now be used to your LIO-way of doing Chetan Loke <English ain't my first language. I don't need any translation because I speak one language - It's, Linux! The language and choice of the GNU generation> --
TCM_Loop and scst_local are exactly comparable and do absolutely the same thing. If you see any difference, please be precise. Internal implementation doesn't matter as soon as it doesn't affect the All those can be done for STGT. LIO isn't needed for that. So, value for STGT users still not questionable to me. Vlad --
On Wed, Aug 18, 2010 at 12:18 PM, James Bottomley Of course it's relevant. Not all engineers know about everything when they venture in a new area. SCST overcomes that. It's not intimidating. Infact, 3 months back, I asked a storport/miniport(aka Windows) guy to take a look at scst and after studying the README he was amazed to see how easy it was. Plus,everyone knows that if a solution is inbox then it's just easier to maintain. Also no need to patch/re-spin my iso. And you are missing the point - find me one good technical conversation thread on LIO! What does that tell you? Why not give a chance(or atleast consider?) to someone who has a wide/active user base? LIO never struggled to push their code upstream and it still is a viable candidate? On the other hand scst users/developers are ready to bend over to accommodate all the changes! What does that tell you? James, how can a community remain vibrant when one solution is favoured over another w/o any clear explanation and justification? It's like the old saying - the lips are moving but all I hear is blah blah blah. We look up to you but why should we accept the outcome of a -- cloke --
On Wed, Aug 18, 2010 at 5:11 PM, James Bottomley Hello James, Thanks for taking notes during the storage track and sharing these notes (http://lwn.net/Articles/400589/). These notes are interesting but do not reveal why LIO is preferred. Also, the list with the three acceptance criteria is incomplete. A very important criterion before any kernel code can be merged upstream is whether or not there is a maintainer for that code. Someone who has proven prior kernel coding experience and someone who understands the new code thoroughly. Bart. --
Quick question ... will LIO support SRP? I should probably run over to their lists and ask, but a quick inspection of their site this morning shows that they are mostly iSCSI and FC focused. (LIO folks, please feel free to step up and comment) I'd certainly like to see a single framework, but not at the cost of losing important (to us) functionality. We'd like to continue to use SRP, and iSCSI. We'd like to use iSER (which is in the tgt stack) which LIO would give us when merged with tgt. We are currently using SCST's iSCSI and SRP stack within our products. I believe this also affects the OFED folks, as SRP is one of their services (based upon SCST). Any guidance (in a general sense on the target side, not necessarily specific to LIO) on this would be appreciated. SCST has been a good system for us to work with. I'd hate to lose its functionality, and have us be forced to re-engineer some of our backend logic to work around the missing bits. Thanks! Regards, Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: landman@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/jackrabbit phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 --
Joe, For iSER LIO would not give you anything SCST can't give you. With LIO you would use the iSER target via STGT pass-through sg/bsg. You can the same way use it with SCST scst_local module. Vlad --
I believe there are no licensing issues with SCST. All the patches I have submitted and going to submit licensed under GPLv2. Vlad --
