login
Search
Recently there has been a lot of publicity about these new "Hybrid Hard Drives" (Hard drives with flash cache), what I don't understand is why the flash-cache must be locate at the hard dive.. I mean a lot of people have UBS2 flash drives laing about, how hard would it be to make a general block cache driver that stores on these? I think the following implementation we do nicely
i = 0
blockread(block_nr)
if block_nr is in cache
return block from cache
write all dirty blocks to disk and mark them as clean
read block_nr from disk into block i on flashdrive
increment i modolo numer of blocks on flashdrive
return block
blockwrite(block_nr,block)
if block_nr is in cache
override block
return
if all blocks are dirty
write all blocks to disk and mark them as clean
i = first non dirty block on flashdrive
write block to block i on flashdrive
mark block i as dirty
Can anyone tell me why this wouldn't work, or if it would, why it is not implemented (and in case it is, slap me on the head for not being able to use google)
And what about the bus speed.
And what about the bus speed. The max bus speed for USB2.0 is <30 MBps. While for IDE it's 133 MBps.
And moreover Hybrid Drives are targetted for laptops to improve on power consumption. To use any USB device the host has to supply it with power. i.e. you would be consuming more power than before.
Speed is not everything
There are some people, that wouldn't mind a drop in speed, this would prolong the battery life of your laptop sins you would hardly ever need to spin up you hard drive (like in laptop_mode but flash memory isn't volatile), also some people might think that their hard drive is noisy and could also live with the performance drop, of cause this is not for everyone, and surly there should be some way of disabling it when doing HDD heavy tasks. The teoreticly speed of USB2 is 60MB/S which is faster then my crappy hdd, though wikipedia sayes that the best speed optainable with current flashdrives is about the half.
Don't know why my last commen
Don't know why my last comment was trimmed ??
But anyways the thing is usb devices derive power from the host itself, so I don't see you saving any power here.
Yes 60 is theoritical speed, the best speed available for usb2.0 flash drive is ~30 MBps while for IDE it's 133MBps (or around that mark) and thats huge. Ok maybe you are ready to take a performance hit, well then that beats the very purpose of a 'cache'. And also I am not sure how many people would be ready to pay the extra $$ , compare $$ per Gig for flash vs magnetic memory and you would know.
"so I don't see you saving an
"so I don't see you saving any power here."
motors obviously consume no power?
Flash Memory for a Hard Drive Cache?
My question would be about how many write cycles it can survive. Maybe they have solved that, because they are getting pretty good read speeds (108 MBytes/sec), and fair write speeds (18 MB/sec).
I see why they are doing this, so that the HDD is off most the time and the computer is running off the flash virtual hard drive, and that would save power. I believe running the HDD motor takes more current than a quiescent flash memory, whether it is in the HDD or in the USB slot.
But for it to work to use a usb flash drive for this application, you would have to have some way to spin down the HDD and use the usb drive for most stuff. I don't think a method that is reliable is readily available, but the hybrid hard drive would have internal software to make that happen. Also, it would be pretty inconvenient to have to have that usb drive sticking out all the time.
Just my thoughts. It would work just about as fast with usb 2.0 as it does in the drive, perhaps well enough, there would just have to be software to reliably spin down the drive so you actually get a power saving effect from no drive motor current, and it would be good if you could use the usb device without it being in danger of getting knocked off.
Best way...
The best way to do this is to have two hard drives, one flash, one spinning IDE. The flash one should hold the OS and all your your most common apps, while your disk drive holds your data that changes. Using some utils you can mount the flash disk RO and then just commit the changes to disk when you do OS updates. This woould also add a layer of security to the system since any viruses that get in would mostly likely be wiped out when you reboot. Some flash drives today can do 4million write cycles and have intelligent write balancing across the memory so one area isn't stressed to failure while other areas go unused.
That's what some transaction servers do
They store their journal on a non-volatile solid-state storage device. The actual filesystem being journaled is on a normal HD. Journals naturally round-robin, spreading the writes evenly over the flash's surface. Thus, they naturally wear-level.
I'd still like a boot cache on the HD, though, that would store the stuff I need up through initial login. That'd speed up my laptop boot time dramatically.
The reason for putting this on the HD and not on a USB key is that Flash on the HD could be potentially much faster and the whole thing could be more transparent to the host OS. The flash could just look like part of the disk's normal cache.
So, my recommendation would be to provide flash on the HD, provide the option to make a "journaling partition" on portion of it, and use the rest for a boot/OS cache. Further, on the journaling partition, buffer the journal in RAM, and push the RAM to flash only on a power failure or if the disk is spun down. Otherwise use the disk to back the Journal. Keep enough backup charge in a capacitor somewhere (and in the rotational inertia of the platters back-driving the servo as a generator) to be able to copy journal-in-RAM to journal-in-flash on a power failure.
Security and Disposal Considerations
There are some interesting security implications of hybrid drives. Flash storage and magnetic media might need different disposal protocols. The platters can be erased magnetically (de-gaussed) while the Flash cannot. Of course, both can be physically crushed. An advantage of keeping the flash separate from the drive is that they can be disposed differently (and at different times due to differing failure rates/modes).
See: http://www.appliedmiscellany.com/blog/archives/34
ACTUAL ANSWER!!
Lol, to actually answer your question: You can't use any old flash drive.
The main issue is write cycles. Lots of the new ones have only 10,000 write cycles. To put that in perspective, lets say you have something that writes to the same file once a minute. You'd run out of write cycles in 7 days. Not quite what you'd want :) There are flash drives out there that offer 5M write cycles, but those are physically bigger. I think this is only now becoming possible because of new flash memory types that offer unlimited write cycles (and not surprisingly are more expensive).
Standard misconceptions about flash
Transfer rate
transfer rate than fast CF/USB flash.
Spin up time
first few seconds of boot up so the hard disk is not needed until
it is ready
Wear leveling
the wear leveling software in the device will change the sector
map, so the flash will not wear out after only a few thousand
writes.
access to physical flash sectors.
Spin up/spid down power reduction:
Journalling flash file systems are not that simple:
not want him to. This may cause the sector currently being
written/erased to contain random data.
flash.
did, that sector would wear out.
journal.
it - a later journal entry may show changes to the data.
on specific physical flash sectors.
writing a sector may cause a change to the sector map - which
causes a write to another sector ...
The example stategy sketched out above has problems - how do
you know if a sector is in the cache? There must be some sort of
map between flash block numbers and disc block numbers. That map
cannot be in ram, or you will have no clue what is going on
after a power cycle. The map cannot be on disk - otherwise any
write to flash causes the hard-disk to spin up to store the
changes in the map. The map has to be in flash.
I think a low power system might benefit from using a large
data cache on flash CF/USB flash. If anyone wants to try it,
remember to compare the results setting the VM system to
laptop mode with lots of RAM.
I think a fast system would benefit from a new filesystem -
keep the meta-data and small files in flash and large sequential
files on disk.
If you care about boot times, use a compressed read only
filesystem like cramfs.
Flash based swap is probably the wrong answer - your mileage will
vary depending on whether your swap access is mostly random or
mostly sequential. Either way, you will get much better results
if you just buy enough RAM.
Nice comment, but. CF/USB
Nice comment, but.
I don't understand. So, does it provide direct/unmapped access or not?
No, a flash device generally
No, a flash device generally does not provide unmapped/direct access. But te JFFS2 filesystem (and similar) require this for their mapping algorithms to work as intended.
Introduction to flash devices
The two main types of flash are NAND and NOR. NAND costs less
per megbyte than NOR, and has a higher capacity per chip but a new
NAND device will probably have some broken sectors and will require
an error correcting code. NOR chips have large sectors - 64KB for
example. NAND chips have much smaller sectors - a small multiple
of 528 is common (The other sixteen bytes are often used for
error correction and mapping). NOR devices are often directly
mapped into memory, so random access is possible (in some devices,
data cannot be read during a program or erase cycle). NAND devices
are normally accessed by a serial protocol. This makes it difficult
(but not impossible) to boot from a NAND device. NAND chips must
be programmed one sector at a time. NOR chips can be programmed
one byte at a time, but as programming takes milliseconds, it is
qucker to program as much as possible (say 32 bytes) in each step.
NAND and NOR devices can erase a complete sectors, and both have
bulk erase commands. NOR devices can only program ones to zeroes -
an erase is required to turn zeroes into ones. Some NAND devices
can change zeroes to ones a few times without an erase command,
but this is becomming rare.
Hard disks have a much simpler command set, and much fewer
restrictions. Compact flash cards pretend to be an IDE controller.
They understand only IDE commands to mapped sectors. There is
no way to dectect whether the flash is NAND or NOR. There is
no way to read the sector size or error correction data. There is
no way to control how IDE sectors map to flash sectors. USB flash
devices pretend to be a SCSI interface with a hard disk attached.
They understand only SCSI disk commands, so they provide the same
sort of access as compact flash.
Some Nand flash controller chips do not pretend to be an IDE or
SCSI interface. They understand flash commands. NOR flash is normally
memory mapped without a controller, so they are only accessible with
flash commands. Journaling flash file system 2 (JFFS2) requires
flash commands to operate. Commands that read the type of flash,
sector sizes, programming time-outs; commands to read write and
erase physical sectors. JFFS2 provides journaling and wear leveling.
It is possible to convert a normal block device into a memory
technology device that JFFS2 requires. This conversion adds the
required limitations, and converts flash commands into block device
read/writes. It is possible to convert a CF or USB flash device
into an MTD device. This gives you all the disadvantages of JFFS2,
but does not circumvent the USB/CF SCSI/IDE emulation. Both the
USB/CF controller and JFFS2 will do their own mapping and wear
leveling.
JFFS2 is a good flash filesystem when there is no IDE/SCSI
emulation. USB and CF devices only provide emulated access, so
JFFS2 work work badly on them. Although the wear leveling and
mapping provided by USB and CF are OK, they cannot make assumptions
about the data the are given. When it is possble to access flash
without IDE/SCSI emulation, a kernel could use its knowledge of
the data to do a better job of these tasks. We could have a
--flash option for mkswap. Filesystems could indicate which pages
are the best candidates for caching in flash. Unfortunately, the
highest bandwidth interfaces are used by hard disks. Flash devices
will pretend to be hard disks so that they can work with inflexible
operating systems and bioses.
remapping
Cool info, thanks.
How does the automatic sector remapping in usb/cf devices work?
So, the device detects a write failure. It remaps the sector and never lets the user know? Does it signal the write operation as completed only after it really has completed? What if it runs out of spare sectors? Does the capacity decrease?
Well, and don't hard drives do similar kind of sector remapping?